Day 13: Multicluster – Failover and Federated Services
Goal: You deliberately distribute traffic across local and remote instances, take a cluster down and measure what the client notices – with the gateway and with a federated service. · Duration: approx. 90 min · Prerequisite: Day 12
Briefing
- A mirrored Service is a normal Service. With an HTTPRoute you can place it next to the local one with weights – traffic shifting across cluster boundaries.
- The controller probes the gateway every 3 s. If it fails, the controller empties the mirror’s endpoints. The client’s proxy is faster: it avoids the dead gateway on its own after about 3 s.
- A federated service
(
<name>-federated) combines Services of the same name from several clusters into one. The proxy distributes directly across all Pods. - Federated means Pod-to-Pod, without a gateway. The Pod networks must be able to reach each other.
- The old failover extension (TrafficSplit) is deprecated. Federated services replace it.
ost west
last ─> echo-federated ─┬─> echo-ost │
└──── direkt ────────┴─> echo-west (10.42.0.0/16)
Drill 1: Two Instances, One Name
echo runs in the namespace tag13 in both
clusters. The Deployments are called echo-ost and
echo-west, so the Pod name tells you who answered:
mkdir -p ~/linkerd/tag13 && cd ~/linkerd/tag13
WEST=172.16.0.182
cat > echo.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: tag13
annotations: {linkerd.io/inject: enabled}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: echo, namespace: tag13}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: echo-CLUSTER, namespace: tag13}
spec:
replicas: 1
selector: {matchLabels: {app: echo}}
template:
metadata: {labels: {app: echo}}
spec:
serviceAccountName: echo
containers:
- name: echo
image: ghcr.io/mccutchen/go-httpbin:2.26.0
env: [{name: USE_REAL_HOSTNAME, value: "true"}]
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: echo, namespace: tag13}
spec:
selector: {app: echo}
ports: [{port: 80, targetPort: http, name: http}]
EOF
for c in ost west; do
sed "s/CLUSTER/$c/" echo.yaml | kubectl --context $c apply -f -
done
kubectl --context west -n tag13 label service echo mirror.linkerd.io/exported=trueThe order is right: the namespace in ost exists before
west exports. The load generator writes only who answered
for each call. The target is in ZIEL so that you can switch
it later:
cat > last.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: last, namespace: tag13}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag13}
spec:
replicas: 1
selector: {matchLabels: {app: last}}
template:
metadata: {labels: {app: last}}
spec:
serviceAccountName: last
containers:
- name: last
image: docker.io/curlimages/curl:8.23.0
env: [{name: ZIEL, value: echo}]
command: [sh, -c]
args:
- |
while true; do
r=$(curl -s -m 2 http://$ZIEL/hostname | grep -o "echo-[a-z]*")
echo "$(date +%T) ${r:-fehler}"
sleep 0.5
done
EOF
kubectl --context ost apply -f last.yaml
kubectl --context ost -n tag13 rollout status deploy/last
kubectl --context ost -n tag13 get svcExpected:
deployment "last" successfully rolled out
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo ClusterIP 10.108.151.187 <none> 80/TCP 11s
echo-west ClusterIP 10.100.226.150 <none> 80/TCP 9s
sleep 30
kubectl --context ost -n tag13 logs deploy/last -c last --since=30s \
| awk '{print $2}' | sort | uniq -cExpected (numbers vary):
45 echo-ost
The baseline: everything stays local. A mirror alone does not redirect any traffic.
Save your progress? Sign up for free
Drill 2: Distributing Traffic Across the Boundary
An HTTPRoute attaches to the Service echo and
distributes across echo and the mirror:
cat > route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: echo, namespace: tag13}
spec:
parentRefs:
- {group: core, kind: Service, name: echo, port: 80}
rules:
- backendRefs:
- {name: echo, port: 80, weight: 50}
- {name: echo-west, port: 80, weight: 50}
EOF
kubectl --context ost apply -f route.yaml
sleep 58
kubectl --context ost -n tag13 logs deploy/last -c last --since=55s \
| awk '{print $2}' | sort | uniq -c
linkerd --context ost viz stat -n tag13 deploy/last --to svc/echo-westExpected (numbers vary):
39 echo-ost
52 echo-west
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 TCP_CONN
last 1/1 100.00% 0.9rps 5ms 10ms 10ms 1
Traffic shifting just like within your own cluster, except that one backend lives elsewhere. This is how you gradually move traffic into a new cluster, for example during a migration.
Save your progress? Sign up for free
Drill 3: west Fails – Gateway Mode
You take away the gateway in west. The Pods there keep
running but are no longer reachable:
kubectl --context west -n linkerd-multicluster scale deploy/linkerd-gateway --replicas=0
date +%T
sleep 20
linkerd --context ost multicluster gateways
kubectl --context ost -n tag13 get endpoints echo-westExpected (abridged):
21:59:39
CLUSTER ALIVE NUM_SVC LATENCY
west False 1 -
NAME ENDPOINTS AGE
echo-west 5m44s
After three failed probes the controller empties the mirror’s
endpoints. What did last see? Replace 21:59:39 with your
time from date:
kubectl --context ost -n tag13 logs deploy/last -c last --since=60s \
| awk '$1>="21:59:39" && $1<="21:59:47"'Expected (abridged):
21:59:39 echo-west
21:59:40 echo-ost
21:59:41 echo-ost
21:59:43 fehler
21:59:44 echo-ost
21:59:45 echo-ost
21:59:47 echo-ost
One error, two seconds of standstill (curl -m 2) – after
that everything goes to ost, even though the route still
says 50 to 50. Was that the empty endpoints? The timestamps in the log
of the last proxy tell a different story:
kubectl --context ost -n tag13 logs deploy/last -c linkerd-proxy --since=60s \
--timestamps | grep -E 'closed|failfast|Removing' | sed -E 's/^.{11}(.{8}).*: /\1 /'Expected:
21:59:41 Service failed error=channel closed
21:59:44 Unavailable; entering failfast timeout=3.0
21:59:48 Removing endpoint addr=172.16.0.182:4143
The order explains the recovery. At 21:59:41 the gateway closes the
connection, and reconnects go nowhere. After 3 s without a reachable
endpoint the proxy puts the backend echo-west into
failfast: the waiting request fails (the one
fehler), and the route distributes only across backends
that are ready. The controller empties the endpoints only at 21:59:48,
about 9 s after the outage (3 probes × 3 s). That is just cleanup – the
proxy itself had already rescued the traffic. Back again:
kubectl --context west -n linkerd-multicluster scale deploy/linkerd-gateway --replicas=1
date +%T
sleep 50
linkerd --context ost multicluster gateways
kubectl --context ost -n tag13 logs deploy/last -c last --since=60s \
| grep -m1 echo-westExpected:
22:00:20
CLUSTER ALIVE NUM_SVC LATENCY
west True 1 8ms
22:00:42 echo-west
A good 20 seconds, although the gateway Pod is ready after about 6 s. The controller’s probes run through its own proxy, and that proxy has remembered the dead address:
kubectl --context ost -n linkerd-multicluster logs deploy/controller-west \
-c linkerd-proxy --since=3m --timestamps \
| grep -E 'Failed to connect|exited failfast' | sed -E 's/^.{11}(.{8}).*: /\1 /'Expected (abridged):
21:59:43 Failed to connect error=connect timed out after 1s
21:59:44 Failed to connect error=connect timed out after 1s
21:59:45 Failed to connect error=connect timed out after 1s
21:59:47 Failed to connect error=connect timed out after 1s
21:59:50 Failed to connect error=connect timed out after 1s
21:59:54 Failed to connect error=connect timed out after 1s
22:00:01 Failed to connect error=connect timed out after 1s
22:00:15 Failed to connect error=connect timed out after 1s
22:00:41 Available; exited failfast elapsed=59.295263099
The gaps grow: 1, 2, 3, 4, 7, 14, 26 seconds (backoff). Only the
attempt at 22:00:41 reaches the gateway; then the probe turns green and
the endpoint returns. The longer west was gone, the longer
the return takes – in the test, a good 40 s after an outage of just over
two minutes; the gap grows up to 60 s.
Save your progress? Sign up for free
Drill 4: ost Fails
Now the opposite direction, still at 50 to 50:
kubectl --context ost -n tag13 scale deploy/echo-ost --replicas=0
sleep 58
kubectl --context ost -n tag13 logs deploy/last -c last --since=55s \
| awk '{print $2}' | sort | uniq -cExpected:
84 echo-west
Not a single error – west takes over. But be careful
with weight 0:
kubectl --context ost -n tag13 patch httproute echo --type=json \
-p '[{"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":0}]'
sleep 40
kubectl --context ost -n tag13 logs deploy/last -c last --since=30s \
| awk '{print $2}' | sort | uniq -c
kubectl --context ost -n tag13 exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code}\n' -m 3 http://echo/hostnameExpected:
48 fehler
503
Weight 0 means “never”, even when nothing else is left. For a standby that only steps in during an emergency, weights are the wrong tool. Back to the initial state:
kubectl --context ost -n tag13 scale deploy/echo-ost --replicas=1
kubectl --context ost -n tag13 rollout status deploy/echo-ost
kubectl --context ost delete -f route.yamlSave your progress? Sign up for free
Drill 5: A Flat Network
Federated services talk to Pods in the other cluster directly. Today that doesn’t work:
kubectl --context ost -n tag13 run ohne --image=docker.io/curlimages/curl:8.23.0 \
--restart=Never --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl --context ost -n tag13 wait --for=condition=Ready pod/ohne
P=$(kubectl --context west -n tag13 get pod -l app=echo \
-o jsonpath='{.items[0].status.podIP}')
echo $P
kubectl --context ost -n tag13 exec ohne -- \
curl -s -m 3 -o /dev/null -w '%{http_code}\n' http://$P:8080/hostnameExpected:
10.42.0.21
000
command terminated with exit code 28
The nodes in ost don’t know 10.42.0.0/16. A static route
on every node is enough for the lab. Cilium masquerades outgoing traffic
to the node IP, so west needs no route back:
sudo ip route add 10.42.0.0/16 via $WEST
for w in 172.16.0.228 172.16.0.83; do
ssh ubuntu@$w sudo ip route add 10.42.0.0/16 via $WEST
done
ip route get $P
kubectl --context ost -n tag13 exec ohne -- \
curl -s -m 3 -o /dev/null -w '%{http_code}\n' http://$P:8080/hostnameExpected:
10.42.0.21 via 172.16.0.182 dev eth0 src 172.16.0.126 uid 1000
cache
200
The routes are not persistent and are gone after a reboot. That is intentional here.
Save your progress? Sign up for free
Drill 6: A Federated Service
The label mirror.linkerd.io/federated=member goes on the
Service in both clusters:
for c in ost west; do
kubectl --context $c -n tag13 label service echo mirror.linkerd.io/federated=member
done
sleep 10
kubectl --context ost -n tag13 get svc
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80Expected:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo ClusterIP 10.108.151.187 <none> 80/TCP 7m13s
echo-federated ClusterIP 10.103.75.249 <none> 80/TCP 11s
echo-west ClusterIP 10.100.226.150 <none> 80/TCP 7m11s
NAMESPACE IP PORT POD SERVICE
tag13 10.0.1.144 8080 echo-ost-555d67ccf8-6hz89 echo.tag13
tag13 10.42.0.21 8080 echo-west-555d67ccf8-c7pcc echo.tag13
One Service, two Pods from two clusters, both with their Pod
IP – no gateway. This works with the existing Link from Day 12.
Now switch last over:
kubectl --context ost -n tag13 set env deploy/last ZIEL=echo-federated
kubectl --context ost -n tag13 rollout status deploy/last
sleep 58
kubectl --context ost -n tag13 logs deploy/last -c last --since=55s \
| awk '{print $2}' | sort | uniq -cExpected (numbers vary):
31 echo-ost
54 echo-west
No route, no weights – the proxy distributes across all endpoints by
latency. And who arrives in west now?
linkerd --context west diagnostics proxy-metrics -n tag13 deploy/echo-west \
| grep '^tcp_open_total{direction="inbound",peer="src"' \
| grep -oE '(target_port|client_id|no_tls_reason)="[^"]*"|[0-9]+$' | paste - - -Expected (abridged):
target_port="8080" client_id="last.tag13.serviceaccount.identity.linkerd.cluster.l…" 1
target_port="8080" client_id="linkerd-gateway.linkerd-multicluster.serviceaccou…" 4
target_port="8080" no_tls_reason="no_tls_from_remote" 52
Through the federated service, echo-west sees the real
client last instead of the gateway (those were the calls
from Drill 2). Without TLS there are only the kubelet’s probes and your
test from Drill 5.
Save your progress? Sign up for free
Drill 7: west Fails Hard
This time not just the gateway: you stop k3s entirely, including API
and Pods. k3s-killall.sh terminates all containers but
leaves other services on the VM running:
date +%T
ssh ubuntu@$WEST sudo k3s-killall.sh > /dev/null 2>&1
sleep 50
kubectl --context ost -n tag13 logs deploy/last -c last --since=55s \
| awk '{print $2}' | sort | uniq -c
linkerd --context ost multicluster gateways
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80Expected:
16:31:33
75 echo-ost
5 echo-west
CLUSTER ALIVE NUM_SVC LATENCY
west False 1 -
NAMESPACE IP PORT POD SERVICE
tag13 10.0.1.144 8080 echo-ost-555d67ccf8-6hz89 echo.tag13
tag13 10.42.0.21 8080 echo-west-555d67ccf8-c7pcc echo.tag13
Not a single fehler. Interesting: the dead endpoint is
still in the list, because the API of west no longer
answers and nobody can remove it. The proxy notices by itself:
kubectl --context ost -n tag13 logs deploy/last -c linkerd-proxy --since=60s \
| grep -o 'endpoint{addr=[0-9.:]*}: linkerd_reconnect: .*' | tail -2Expected (abridged):
endpoint{addr=10.42.0.21:8080}: linkerd_reconnect: Failed to connect error=connect …
endpoint{addr=10.42.0.21:8080}: linkerd_reconnect: Failed to connect error=connect …
An endpoint to which no connection can be opened receives no
requests. The proxy keeps trying in the background with increasing
intervals. Now bring west back:
date +%T
ssh ubuntu@$WEST sudo systemctl start k3s
kubectl --context west wait --for=condition=Ready node/west --timeout=120s
kubectl --context west -n tag13 get pod -o wideExpected (abridged):
16:33:07
node/west condition met
NAME READY STATUS RESTARTS AGE IP ...
echo-west-555d67ccf8-c7pcc 2/2 Running 2 (101s ago) 11m 10.42.0.10 ...
The Pod has a new IP. It takes a while until
ost knows it and sends traffic:
sleep 120
kubectl --context ost -n tag13 logs deploy/last -c last --since=300s \
| awk '$1>"16:33:00" && $2=="echo-west"' | head -1Expected:
16:35:16 echo-west
A good two minutes after k3s started: API up, Pods recreated, the
destination controller’s watch reconnected. Throughout the entire outage
there was no error – but also no traffic to west for two
minutes.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show from which remote cluster
echo-federatedhas members. Target time: 30 s - Show the endpoints of
echo-federated. Target time: 20 s - Show why a client without a proxy cannot use
echo-federated. Target time: 45 s - Show the proxy warnings of
lastabout unreachable endpoints. Target time: 30 s
Solution
kubectl --context ost -n tag13 get svc echo-federated \
-o jsonpath='{.metadata.annotations.multicluster\.linkerd\.io/remote-discovery}'
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80
kubectl --context ost -n tag13 get endpointslices \
-l kubernetes.io/service-name=echo-federated
kubectl --context ost -n tag13 exec ohne -- \
curl -s -m 3 -o /dev/null -w '%{http_code}\n' http://echo-federated/hostname
kubectl --context ost -n tag13 logs deploy/last -c linkerd-proxy | grep reconnectExpected (abridged):
echo@west
...
No resources found in tag13 namespace.
000
command terminated with exit code 7
echo-federated has no selector and no EndpointSlices.
Only Linkerd’s destination controller knows the members – no proxy, no
target.
Save your progress? Sign up for free
Checkpoint
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80Expected: two lines, one with 10.0.x.x
(echo-ost), one with 10.42.x.x
(echo-west).
kubectl --context ost -n tag13 logs deploy/last -c last --since=30s \
| awk '{print $2}' | sort | uniq -cExpected: lines for echo-ost and
echo-west, none for fehler.
linkerd --context ost multicluster check 2>&1 | tail -1Expected:
Status check results are √
Challenge
Turn the mirror echo-west into a mirror without
a gateway: it should point directly to the Pod in
west, still under the name echo-west.
Hint: Besides exported=true, the Link
knows a second value for the export label.
Solution
kubectl --context west -n tag13 label service echo \
mirror.linkerd.io/exported=remote-discovery --overwrite
sleep 10
kubectl --context ost -n tag13 get svc echo-west \
-o jsonpath='{.metadata.labels.multicluster\.linkerd\.io/remote-discovery}{"\n"}'
linkerd --context ost diagnostics endpoints echo-west.tag13.svc.cluster.local:80
kubectl --context ost -n tag13 exec deploy/last -c last -- \
curl -s http://echo-west/hostnameExpected (abridged):
west
NAMESPACE IP PORT POD SERVICE
tag13 10.42.0.10 8080 echo-west-555d67ccf8-c7pcc echo.tag13
{
"hostname": "echo-west-555d67ccf8-c7pcc"
}
The Link from Day 12 already contains
remoteDiscoverySelector
(mirror.linkerd.io/exported=remote-discovery). The mirror
stays but points to the Pod IP. As with the federated service, this
requires a flat network.
Save your progress? Sign up for free
Cleanup
First the exports, then the Link. Since Linkerd 2.18 there is no
multicluster unlink anymore; you remove the parts one by
one:
kubectl --context ost -n tag13 delete pod ohne
kubectl --context west -n tag13 label service echo \
mirror.linkerd.io/exported- mirror.linkerd.io/federated-
kubectl --context ost -n tag13 label service echo mirror.linkerd.io/federated-
kubectl --context ost get svc -A -l mirror.linkerd.io/mirrored-service=true
kubectl --context ost -n linkerd-multicluster delete link west
linkerd --context ost multicluster install -f ../tag12/mc-ost-gw.yaml \
| kubectl --context ost delete -l mirror.linkerd.io/cluster-name=west -f -
kubectl --context ost -n linkerd-multicluster delete secret cluster-credentials-west
kubectl --context ost -n linkerd delete secret cluster-credentials-westExpected (abridged):
No resources found
link.multicluster.linkerd.io "west" deleted from linkerd-multicluster namespace
deployment.apps "controller-west" deleted from linkerd-multicluster namespace
service "probe-west" deleted from linkerd-multicluster namespace
...
Then routes and namespaces:
sudo ip route del 10.42.0.0/16 via $WEST
for w in 172.16.0.228 172.16.0.83; do
ssh ubuntu@$w sudo ip route del 10.42.0.0/16 via $WEST
done
for c in ost west; do kubectl --context $c delete namespace tag13; done
linkerd --context ost multicluster check 2>&1 | tail -1
linkerd --context ost multicluster gatewaysExpected:
Status check results are √
CLUSTER ALIVE NUM_SVC LATENCY
The extension stays installed on both clusters, and west
keeps running. You can create a new Link at any time: controller on
ost with ../tag12/mc-ost-gw.yaml (Day 12,
Drill 5 and Challenge – with mc-ost-values.yaml the
NodePort gateway would disappear), then link-gen as in
Drill 6.
Interview Note
- How do you do failover between clusters with Linkerd? Federated services: the proxy distributes across all Pods of all clusters and avoids endpoints that don’t respond. The old failover extension with TrafficSplit is deprecated.
- Gateway or Pod-to-Pod? The gateway only needs a reachable address, costs one hop and hides the client identity. Pod-to-Pod needs routed, non-overlapping Pod networks, but keeps the real identity.
- What happens when an entire cluster is gone? Its endpoints remain known because its API doesn’t respond. The proxy can’t get through and stops sending anything there.
Enterprise Note
- In production, flat networks come from the network design (BGP, Cilium ClusterMesh, cloud VPC peering), not from static routes.
- Federated services need meshed clients. Ingress controllers and Jobs
without a proxy see
-federatedas an empty Service. - Identical Service names in all clusters are mandatory for federation – define naming conventions up front.
- Practice outages regularly (game days): gateway gone, API gone, entire cluster gone. Recovery takes longer than the outage.
- Policies: through the gateway the server sees the gateway identity, through Pod-to-Pod the real client. Write rules for both paths.