Day 9: Traffic Shifting
Goal: You shift traffic step by step from an old version to a new one, let testers reach the new version on purpose via a header, and roll back in seconds. · Duration: approx. 60 min · Prerequisite: Day 8
Briefing
- An HTTPRoute with a Service as its
parentRefapplies to all clients that call this Service. ItsbackendRefswithweightdistribute the requests. - The client proxy decides. Clients without a proxy do not notice the route at all.
- The Service remains the stable address. No client and no Pod is changed, no restart – weights take effect immediately.
- Instead of chance, a header can decide: dark launch for testers.
- Today in
tag09with a separate copy ofnotes-apiin two versions, without a database.
HTTPRoute notes-api-canary
last ─> proxy ──┬── 90 % ──> Service notes-api ──> notes-api-v1 (0.5.2)
└── 10 % ──> Service notes-api-v2 ──> notes-api-v2 (0.7.1)
Drill 1: Two versions side by side
Without NOTES_DB_URL, notes-api starts with
an in-memory store and reports its version under /version –
exactly right for today.
mkdir -p ~/linkerd/tag09 && cd ~/linkerd/tag09
kubectl create namespace tag09
kubectl annotate namespace tag09 linkerd.io/inject=enabled
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: notes-api, namespace: tag09}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: last, namespace: tag09}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api-v1, namespace: tag09}
spec:
replicas: 1
selector: {matchLabels: {app: notes-api, version: v1}}
template:
metadata: {labels: {app: notes-api, version: v1}}
spec:
serviceAccountName: notes-api
containers:
- name: notes-api
image: ghcr.io/thomaszachmann/notes-api:0.5.2
env: [{name: NOTES_API_KEY, value: lab-key-123}]
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /healthz, port: http}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api-v2, namespace: tag09}
spec:
replicas: 1
selector: {matchLabels: {app: notes-api, version: v2}}
template:
metadata: {labels: {app: notes-api, version: v2}}
spec:
serviceAccountName: notes-api
containers:
- name: notes-api
image: ghcr.io/thomaszachmann/notes-api:0.7.1
env: [{name: NOTES_API_KEY, value: lab-key-123}]
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag09}
spec:
selector: {app: notes-api, version: v1}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: v1
kind: Service
metadata: {name: notes-api-v2, namespace: tag09}
spec:
selector: {app: notes-api, version: v2}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag09}
spec:
replicas: 1
selector: {matchLabels: {app: last}}
template:
metadata: {labels: {app: last}}
spec:
serviceAccountName: last
containers:
- name: last
image: docker.io/curlimages/curl:8.23.0
env: [{name: URL, value: http://notes-api/version}]
command: [sh, -c]
args:
- |
while true; do
code=$(curl -s -m 5 -o /tmp/v -w '%{http_code}' $URL)
echo "$(date +%T) $(cat /tmp/v) $code"
sleep 0.5
done
EOF
kubectl apply -f app.yaml
kubectl -n tag09 rollout status deploy --timeout=180s
kubectl -n tag09 get pods
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -cExpected (abridged, numbers vary):
NAME READY STATUS RESTARTS AGE
last-7dc884997f-c87hf 2/2 Running 0 9s
notes-api-v1-7d9976cbdc-hc8jx 2/2 Running 0 9s
notes-api-v2-78bcbb6547-h9ms6 2/2 Running 0 9s
92 0.5.2 200
The Service notes-api only selects
version: v1. v2 is running but gets no
traffic. The columns in the log: time, version, status code.
Save your progress? Sign up for free
Drill 2: Canary with ten percent
cat > route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-canary, namespace: tag09}
spec:
parentRefs:
- {group: "", kind: Service, name: notes-api, port: 80}
rules:
- backendRefs:
- {name: notes-api, port: 80, weight: 90}
- {name: notes-api-v2, port: 80, weight: 10}
EOF
kubectl apply -f route.yaml
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n tag09 deploy --from deploy/lastExpected (numbers vary):
78 0.5.2 200
9 0.7.1 200
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 ...
notes-api-v1 1/1 100.00% 1.4rps 4ms 65ms 93ms ...
notes-api-v2 1/1 100.00% 0.2rps 20ms 39ms 40ms ...
The Service notes-api appears in the route as the parent
and as a backend. As a backend it means: its own
endpoints, that is, v1. Without it in
backendRefs, everything would go to v2.
stat --from deploy/last shows the distribution from the
client’s point of view – with the success rate per version, which is
your decision criterion for the next step. group: "" is the
core API group – Linkerd’s docs and the other days write it as
group: core; both mean the same thing.
Save your progress? Sign up for free
Drill 3: Who is actually splitting here?
kubectl -n tag09 get httproute notes-api-canary \
-o jsonpath='{range .status.parents[*]}{.controllerName}{"\n"}{end}'
kubectl -n tag09 get ciliumenvoyconfig
kubectl -n tag09 get endpointslicesExpected (abridged):
io.cilium/gateway-controller
linkerd.io/policy-controller
NAME AGE
notes-api 66s
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
cilium-gateway-notes-api IPv4 9999 192.192.192.192 66s
notes-api-... IPv4 8080 10.0.1.215 3m
Two controllers have accepted the route. Cilium also supports Service
parents (GAMMA, the mesh profile of the Gateway API) and forwards
traffic to the ClusterIP of notes-api through its Envoy;
for this it creates the CiliumEnvoyConfig and an
EndpointSlice with the placeholder address
192.192.192.192:9999. So who really splits? A client
without a proxy versus one with a proxy:
kubectl -n tag09 run ohne --image=curlimages/curl:8.23.0 --restart=Never \
--annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl -n tag09 wait --for=condition=Ready pod/ohne --timeout=60s
kubectl -n tag09 exec ohne -- sh -c 'for i in $(seq 50); do
curl -s -i http://notes-api/version | grep -iE "^server|^0\."; done' | sort | uniq -c
kubectl -n tag09 exec deploy/last -c last -- sh -c 'for i in $(seq 50); do
curl -s -i http://notes-api/version | grep -iE "^server|^0\."; done' | sort | uniq -c
kubectl -n tag09 delete pod ohne --nowExpected (numbers vary):
47 0.5.2
3 0.7.1
50 server: envoy
44 0.5.2
6 0.7.1
Without a proxy, Cilium’s Envoy splits (server: envoy).
With a proxy, Linkerd splits: the proxy resolves the Service itself and
connects directly to the Pod IPs, bypassing the ClusterIP – Cilium’s
rule never applies. Here the result happens to match. As soon as you use
Linkerd-specific features such as retries or timeouts in the route,
clients without a proxy get something different.
Warning: Two GAMMA implementations in one cluster are a trap. Check
status.parentsfor every route and decide on one mesh that implements Service routes.
Save your progress? Sign up for free
Drill 4: 50/50, then everything
You change weights directly in the route, without a file:
W=/spec/rules/0/backendRefs
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
"[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":50},
{\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":50}]"
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -cExpected (numbers vary):
50 0.5.2 200
40 0.7.1 200
On to 0 to 100:
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
"[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":0},
{\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":100}]"
kubectl -n tag09 get httproute notes-api-canary \
-o jsonpath='{.spec.rules[*].backendRefs[*].weight}{"\n"}'
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -cExpected:
0 100
92 0.7.1 200
No Pod has restarted, no request has failed.
Save your progress? Sign up for free
Drill 5: Rollback in seconds
Suppose v2 fails at 50 percent. Back to 50/50, then the
retreat:
sed 's/weight: 90/weight: 50/; s/weight: 10/weight: 50/' route.yaml \
| kubectl apply -f -
sleep 30
date +%T
kubectl -n tag09 delete httproute notes-api-canary
sleep 5
kubectl -n tag09 logs deploy/last -c last --since=8sExpected (abridged):
16:23:21
httproute.gateway.networking.k8s.io "notes-api-canary" deleted from tag09 namespace
16:23:19 0.7.1 200
16:23:19 0.5.2 200
16:23:21 0.5.2 200
16:23:22 0.5.2 200
16:23:22 0.5.2 200
Without a route, the Service itself applies again – so only
v1. The change reaches all client proxies in less than a
second. You can only pull off a clean rollback by deletion or by weight
0 if v1 is still running: scale down the old
version only once the new one is done.
Tip: If
v2fails completely (no endpoints), Linkerd removes the backend from the distribution. In the test, scaling down produced exactly one504, then onlyv1. This is no substitute for a rollback: a faulty but running version keeps getting its share.
Save your progress? Sign up for free
Drill 6: Dark launch via header
At first, only testers should see v2; everyone else
stays on v1:
cat > route-header.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-dark, namespace: tag09}
spec:
parentRefs:
- {group: "", kind: Service, name: notes-api, port: 80}
rules:
- matches:
- headers: [{name: x-canary, value: "true"}]
backendRefs:
- {name: notes-api-v2, port: 80}
- backendRefs:
- {name: notes-api, port: 80}
EOF
kubectl apply -f route-header.yaml
for h in "x-canary: true" "x-canary: false" "X-Canary: true"; do
echo "$h -> $(kubectl -n tag09 exec deploy/last -c last -- \
curl -s -H "$h" http://notes-api/version)"
doneExpected:
httproute.gateway.networking.k8s.io/notes-api-dark created
x-canary: true -> 0.7.1
x-canary: false -> 0.5.2
X-Canary: true -> 0.7.1
Header names are case-insensitive, values are not. The load generator
sends no header and stays entirely on v1. The client proxy
counts which rule a request followed per route and backend – since the
Pod started, which is why the route got a new name:
linkerd diagnostics proxy-metrics -n tag09 deploy/last \
| grep ^outbound_http_route_backend_requests_total | grep notes-api-dark \
| grep -oE 'backend_name="[^"]+"|[0-9]+$' | paste -d ' ' - -Expected (numbers vary):
backend_name="notes-api-v2" 2
backend_name="notes-api" 5
Two requests with the header went to v2, everything else
to notes-api. The second number keeps growing with the load
generator, the first does not.
Important: headers only steer if they reach the caller’s proxy. If a
browser calls web and web calls
notes-api, web has to pass the header on –
nginx does this by itself, many frameworks do not.
Save your progress? Sign up for free
Drill 7: Wrapping up
After a successful rollout, the state belongs back in the Service. Otherwise your production hangs permanently on a route with a weight of 0 to 100:
W=/spec/rules/0/backendRefs
kubectl -n tag09 delete httproute notes-api-dark
kubectl apply -f route.yaml
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
"[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":0},
{\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":100}]"
kubectl -n tag09 patch service notes-api \
-p '{"spec":{"selector":{"version":"v2"}}}'
kubectl -n tag09 delete httproute notes-api-canary
kubectl -n tag09 scale deploy/notes-api-v1 --replicas=0
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag09 get ciliumenvoyconfig
kubectl -n tag09 get endpointslices -l io.cilium.gateway/owning-gateway=notes-api
kubectl -n tag09 delete endpointslice -l io.cilium.gateway/owning-gateway=notes-apiExpected:
112 0.7.1 200
No resources found in tag09 namespace.
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
cilium-gateway-notes-api IPv4 9999 192.192.192.192 11m
endpointslice.discovery.k8s.io "cilium-gateway-notes-api" deleted from tag09 namespace
The order matters: first the Service points to v2, then
the route disappears, then v1 goes. No error in the log.
Cilium’s Envoy configuration disappeared along with the route, but its
EndpointSlice did not – you clean that up by hand. It does no harm (it
belongs to no existing Service), but it is confusing during
troubleshooting.
Save your progress? Sign up for free
Exercise: Speed Drills
Bring v1 back first:
kubectl -n tag09 scale deploy/notes-api-v1 --replicas=1
kubectl -n tag09 patch service notes-api \
-p '{"spec":{"selector":{"version":"v1"}}}'- Create the route from
route.yamland show which controllers have accepted it. Target time: 30 s - Set the route to 80/20 with one command. Target time: 60 s
- Show the distribution of the last minute from the point of view of
lastwithviz. Target time: 15 s - Send a request with
x-canary: truefromlastand show the version. Target time: 20 s
Solution
kubectl apply -f route.yaml
kubectl -n tag09 get httproute notes-api-canary \
-o jsonpath='{.status.parents[*].controllerName}{"\n"}'
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
"[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":80},
{\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":20}]"
linkerd viz stat -n tag09 deploy --from deploy/last
kubectl -n tag09 exec deploy/last -c last -- \
curl -s -H 'x-canary: true' http://notes-api/versionExpected (excerpt):
linkerd.io/policy-controller io.cilium/gateway-controller
0.7.1
For task 4 you sometimes get 0.5.2, sometimes
0.7.1: route.yaml knows no header, only the
weight counts. The Challenge shows dark launch and weights at the same
time.
Save your progress? Sign up for free
Checkpoint
kubectl -n tag09 get podsExpected: last,
notes-api-v1 and notes-api-v2 each
2/2 Running.
linkerd viz stat -n tag09 deploy --from deploy/lastExpected: two lines notes-api-v1 and
notes-api-v2, both 100.00%, RPS in the ratio
of the weights.
kubectl -n tag09 get httproute notes-api-canary \
-o jsonpath='{.status.parents[*].conditions[?(@.type=="Accepted")].status}{"\n"}'Expected: True True – Linkerd and
Cilium have accepted the route.
Challenge
Build a route with which testers always get v2 via
x-canary: true, while everyone else is distributed 90/10.
Prove both.
Hint: A route may have several rules; the one with
matches wins.
Solution
cat > route-challenge.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-canary, namespace: tag09}
spec:
parentRefs:
- {group: "", kind: Service, name: notes-api, port: 80}
rules:
- matches:
- headers: [{name: x-canary, value: "true"}]
backendRefs:
- {name: notes-api-v2, port: 80}
- backendRefs:
- {name: notes-api, port: 80, weight: 90}
- {name: notes-api-v2, port: 80, weight: 10}
EOF
kubectl apply -f route-challenge.yaml
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
for i in 1 2 3 4 5; do
kubectl -n tag09 exec deploy/last -c last -- \
curl -s -H 'x-canary: true' http://notes-api/version
done | sort | uniq -cExpected (numbers vary):
79 0.5.2 200
12 0.7.1 200
5 0.7.1
Save your progress? Sign up for free
Cleanup
kubectl delete namespace tag09Cilium’s EndpointSlices disappear along with the namespace. If you
deleted routes with a Service parent in other namespaces, check there
with
kubectl get endpointslices -A | grep 192.192.192.192.
Interview Note
- How does Linkerd do a canary? With a Gateway API HTTPRoute
whose parent is the Service;
backendRefswith weights distribute in the client proxy. Older guides use the SMI resourceTrafficSplit– for anything new, you use HTTPRoute. - Why does a client without a proxy not see the route? The caller’s outbound proxy implements the route. Without a proxy, only the Service applies.
- How do you roll back? Set the weight of the new version to 0 or delete the route – takes effect immediately, without a restart, as long as the old version is still running.
Enterprise Note
- By hand, this is a learning tool. In production, Flagger or Argo Rollouts (with the Gateway API plugin) shifts the weights of the same HTTPRoute, measures the success rate from Prometheus, and rolls back automatically on deviation.
- A GitOps tool must then not overwrite the weights – ignore the fields or leave the route entirely to the rollout tool.
- Only one GAMMA implementation per Service: if Cilium with Gateway
API runs alongside, it also accepts routes with a Service parent – check
status.parentsof every route. - Header routing requires header propagation through all services in the chain – as with tracing.
- Canary weights are request shares, not users: for “a user stays on one version”, you need headers or cookies.