Skip to content
Hands-on IT IT Drills

Day 9: Traffic Shifting

Goal: You shift traffic step by step from an old version to a new one, let testers reach the new version on purpose via a header, and roll back in seconds. · Duration: approx. 60 min · Prerequisite: Day 8

Briefing

  • An HTTPRoute with a Service as its parentRef applies to all clients that call this Service. Its backendRefs with weight distribute the requests.
  • The client proxy decides. Clients without a proxy do not notice the route at all.
  • The Service remains the stable address. No client and no Pod is changed, no restart – weights take effect immediately.
  • Instead of chance, a header can decide: dark launch for testers.
  • Today in tag09 with a separate copy of notes-api in two versions, without a database.
                      HTTPRoute notes-api-canary
 last ─> proxy ──┬── 90 % ──> Service notes-api    ──> notes-api-v1 (0.5.2)
                 └── 10 % ──> Service notes-api-v2 ──> notes-api-v2 (0.7.1)

Drill 1: Two versions side by side

Without NOTES_DB_URL, notes-api starts with an in-memory store and reports its version under /version – exactly right for today.

mkdir -p ~/linkerd/tag09 && cd ~/linkerd/tag09
kubectl create namespace tag09
kubectl annotate namespace tag09 linkerd.io/inject=enabled
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: notes-api, namespace: tag09}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: last, namespace: tag09}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api-v1, namespace: tag09}
spec:
  replicas: 1
  selector: {matchLabels: {app: notes-api, version: v1}}
  template:
    metadata: {labels: {app: notes-api, version: v1}}
    spec:
      serviceAccountName: notes-api
      containers:
      - name: notes-api
        image: ghcr.io/thomaszachmann/notes-api:0.5.2
        env: [{name: NOTES_API_KEY, value: lab-key-123}]
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /healthz, port: http}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api-v2, namespace: tag09}
spec:
  replicas: 1
  selector: {matchLabels: {app: notes-api, version: v2}}
  template:
    metadata: {labels: {app: notes-api, version: v2}}
    spec:
      serviceAccountName: notes-api
      containers:
      - name: notes-api
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        env: [{name: NOTES_API_KEY, value: lab-key-123}]
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag09}
spec:
  selector: {app: notes-api, version: v1}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: v1
kind: Service
metadata: {name: notes-api-v2, namespace: tag09}
spec:
  selector: {app: notes-api, version: v2}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag09}
spec:
  replicas: 1
  selector: {matchLabels: {app: last}}
  template:
    metadata: {labels: {app: last}}
    spec:
      serviceAccountName: last
      containers:
      - name: last
        image: docker.io/curlimages/curl:8.23.0
        env: [{name: URL, value: http://notes-api/version}]
        command: [sh, -c]
        args:
        - |
          while true; do
            code=$(curl -s -m 5 -o /tmp/v -w '%{http_code}' $URL)
            echo "$(date +%T) $(cat /tmp/v) $code"
            sleep 0.5
          done
EOF
kubectl apply -f app.yaml
kubectl -n tag09 rollout status deploy --timeout=180s
kubectl -n tag09 get pods
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (abridged, numbers vary):

NAME                            READY   STATUS    RESTARTS   AGE
last-7dc884997f-c87hf           2/2     Running   0          9s
notes-api-v1-7d9976cbdc-hc8jx   2/2     Running   0          9s
notes-api-v2-78bcbb6547-h9ms6   2/2     Running   0          9s
     92 0.5.2 200

The Service notes-api only selects version: v1. v2 is running but gets no traffic. The columns in the log: time, version, status code.

Save your progress? Sign up for free

Drill 2: Canary with ten percent

cat > route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-canary, namespace: tag09}
spec:
  parentRefs:
  - {group: "", kind: Service, name: notes-api, port: 80}
  rules:
  - backendRefs:
    - {name: notes-api, port: 80, weight: 90}
    - {name: notes-api-v2, port: 80, weight: 10}
EOF
kubectl apply -f route.yaml
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n tag09 deploy --from deploy/last

Expected (numbers vary):

     78 0.5.2 200
      9 0.7.1 200
NAME           MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   ...
notes-api-v1      1/1   100.00%   1.4rps           4ms          65ms          93ms   ...
notes-api-v2      1/1   100.00%   0.2rps          20ms          39ms          40ms   ...

The Service notes-api appears in the route as the parent and as a backend. As a backend it means: its own endpoints, that is, v1. Without it in backendRefs, everything would go to v2. stat --from deploy/last shows the distribution from the client’s point of view – with the success rate per version, which is your decision criterion for the next step. group: "" is the core API group – Linkerd’s docs and the other days write it as group: core; both mean the same thing.

Save your progress? Sign up for free

Drill 3: Who is actually splitting here?

kubectl -n tag09 get httproute notes-api-canary \
  -o jsonpath='{range .status.parents[*]}{.controllerName}{"\n"}{end}'
kubectl -n tag09 get ciliumenvoyconfig
kubectl -n tag09 get endpointslices

Expected (abridged):

io.cilium/gateway-controller
linkerd.io/policy-controller
NAME        AGE
notes-api   66s
NAME                       ADDRESSTYPE   PORTS   ENDPOINTS         AGE
cilium-gateway-notes-api   IPv4          9999    192.192.192.192   66s
notes-api-...              IPv4          8080    10.0.1.215        3m

Two controllers have accepted the route. Cilium also supports Service parents (GAMMA, the mesh profile of the Gateway API) and forwards traffic to the ClusterIP of notes-api through its Envoy; for this it creates the CiliumEnvoyConfig and an EndpointSlice with the placeholder address 192.192.192.192:9999. So who really splits? A client without a proxy versus one with a proxy:

kubectl -n tag09 run ohne --image=curlimages/curl:8.23.0 --restart=Never \
  --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl -n tag09 wait --for=condition=Ready pod/ohne --timeout=60s
kubectl -n tag09 exec ohne -- sh -c 'for i in $(seq 50); do
  curl -s -i http://notes-api/version | grep -iE "^server|^0\."; done' | sort | uniq -c
kubectl -n tag09 exec deploy/last -c last -- sh -c 'for i in $(seq 50); do
  curl -s -i http://notes-api/version | grep -iE "^server|^0\."; done' | sort | uniq -c
kubectl -n tag09 delete pod ohne --now

Expected (numbers vary):

     47 0.5.2
      3 0.7.1
     50 server: envoy
     44 0.5.2
      6 0.7.1

Without a proxy, Cilium’s Envoy splits (server: envoy). With a proxy, Linkerd splits: the proxy resolves the Service itself and connects directly to the Pod IPs, bypassing the ClusterIP – Cilium’s rule never applies. Here the result happens to match. As soon as you use Linkerd-specific features such as retries or timeouts in the route, clients without a proxy get something different.

Warning: Two GAMMA implementations in one cluster are a trap. Check status.parents for every route and decide on one mesh that implements Service routes.

Save your progress? Sign up for free

Drill 4: 50/50, then everything

You change weights directly in the route, without a file:

W=/spec/rules/0/backendRefs
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
  "[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":50},
    {\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":50}]"
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (numbers vary):

     50 0.5.2 200
     40 0.7.1 200

On to 0 to 100:

kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
  "[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":0},
    {\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":100}]"
kubectl -n tag09 get httproute notes-api-canary \
  -o jsonpath='{.spec.rules[*].backendRefs[*].weight}{"\n"}'
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected:

0 100
     92 0.7.1 200

No Pod has restarted, no request has failed.

Save your progress? Sign up for free

Drill 5: Rollback in seconds

Suppose v2 fails at 50 percent. Back to 50/50, then the retreat:

sed 's/weight: 90/weight: 50/; s/weight: 10/weight: 50/' route.yaml \
  | kubectl apply -f -
sleep 30
date +%T
kubectl -n tag09 delete httproute notes-api-canary
sleep 5
kubectl -n tag09 logs deploy/last -c last --since=8s

Expected (abridged):

16:23:21
httproute.gateway.networking.k8s.io "notes-api-canary" deleted from tag09 namespace
16:23:19 0.7.1 200
16:23:19 0.5.2 200
16:23:21 0.5.2 200
16:23:22 0.5.2 200
16:23:22 0.5.2 200

Without a route, the Service itself applies again – so only v1. The change reaches all client proxies in less than a second. You can only pull off a clean rollback by deletion or by weight 0 if v1 is still running: scale down the old version only once the new one is done.

Tip: If v2 fails completely (no endpoints), Linkerd removes the backend from the distribution. In the test, scaling down produced exactly one 504, then only v1. This is no substitute for a rollback: a faulty but running version keeps getting its share.

Save your progress? Sign up for free

Drill 6: Dark launch via header

At first, only testers should see v2; everyone else stays on v1:

cat > route-header.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-dark, namespace: tag09}
spec:
  parentRefs:
  - {group: "", kind: Service, name: notes-api, port: 80}
  rules:
  - matches:
    - headers: [{name: x-canary, value: "true"}]
    backendRefs:
    - {name: notes-api-v2, port: 80}
  - backendRefs:
    - {name: notes-api, port: 80}
EOF
kubectl apply -f route-header.yaml
for h in "x-canary: true" "x-canary: false" "X-Canary: true"; do
  echo "$h -> $(kubectl -n tag09 exec deploy/last -c last -- \
    curl -s -H "$h" http://notes-api/version)"
done

Expected:

httproute.gateway.networking.k8s.io/notes-api-dark created
x-canary: true -> 0.7.1
x-canary: false -> 0.5.2
X-Canary: true -> 0.7.1

Header names are case-insensitive, values are not. The load generator sends no header and stays entirely on v1. The client proxy counts which rule a request followed per route and backend – since the Pod started, which is why the route got a new name:

linkerd diagnostics proxy-metrics -n tag09 deploy/last \
  | grep ^outbound_http_route_backend_requests_total | grep notes-api-dark \
  | grep -oE 'backend_name="[^"]+"|[0-9]+$' | paste -d ' ' - -

Expected (numbers vary):

backend_name="notes-api-v2" 2
backend_name="notes-api" 5

Two requests with the header went to v2, everything else to notes-api. The second number keeps growing with the load generator, the first does not.

Important: headers only steer if they reach the caller’s proxy. If a browser calls web and web calls notes-api, web has to pass the header on – nginx does this by itself, many frameworks do not.

Save your progress? Sign up for free

Drill 7: Wrapping up

After a successful rollout, the state belongs back in the Service. Otherwise your production hangs permanently on a route with a weight of 0 to 100:

W=/spec/rules/0/backendRefs
kubectl -n tag09 delete httproute notes-api-dark
kubectl apply -f route.yaml
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
  "[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":0},
    {\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":100}]"
kubectl -n tag09 patch service notes-api \
  -p '{"spec":{"selector":{"version":"v2"}}}'
kubectl -n tag09 delete httproute notes-api-canary
kubectl -n tag09 scale deploy/notes-api-v1 --replicas=0
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag09 get ciliumenvoyconfig
kubectl -n tag09 get endpointslices -l io.cilium.gateway/owning-gateway=notes-api
kubectl -n tag09 delete endpointslice -l io.cilium.gateway/owning-gateway=notes-api

Expected:

    112 0.7.1 200
No resources found in tag09 namespace.
NAME                       ADDRESSTYPE   PORTS   ENDPOINTS         AGE
cilium-gateway-notes-api   IPv4          9999    192.192.192.192   11m
endpointslice.discovery.k8s.io "cilium-gateway-notes-api" deleted from tag09 namespace

The order matters: first the Service points to v2, then the route disappears, then v1 goes. No error in the log. Cilium’s Envoy configuration disappeared along with the route, but its EndpointSlice did not – you clean that up by hand. It does no harm (it belongs to no existing Service), but it is confusing during troubleshooting.

Save your progress? Sign up for free

Exercise: Speed Drills

Bring v1 back first:

kubectl -n tag09 scale deploy/notes-api-v1 --replicas=1
kubectl -n tag09 patch service notes-api \
  -p '{"spec":{"selector":{"version":"v1"}}}'
  1. Create the route from route.yaml and show which controllers have accepted it. Target time: 30 s
  2. Set the route to 80/20 with one command. Target time: 60 s
  3. Show the distribution of the last minute from the point of view of last with viz. Target time: 15 s
  4. Send a request with x-canary: true from last and show the version. Target time: 20 s

Solution

kubectl apply -f route.yaml
kubectl -n tag09 get httproute notes-api-canary \
  -o jsonpath='{.status.parents[*].controllerName}{"\n"}'
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
  "[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":80},
    {\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":20}]"
linkerd viz stat -n tag09 deploy --from deploy/last
kubectl -n tag09 exec deploy/last -c last -- \
  curl -s -H 'x-canary: true' http://notes-api/version

Expected (excerpt):

linkerd.io/policy-controller io.cilium/gateway-controller
0.7.1

For task 4 you sometimes get 0.5.2, sometimes 0.7.1: route.yaml knows no header, only the weight counts. The Challenge shows dark launch and weights at the same time.

Save your progress? Sign up for free

Checkpoint

kubectl -n tag09 get pods

Expected: last, notes-api-v1 and notes-api-v2 each 2/2 Running.

linkerd viz stat -n tag09 deploy --from deploy/last

Expected: two lines notes-api-v1 and notes-api-v2, both 100.00%, RPS in the ratio of the weights.

kubectl -n tag09 get httproute notes-api-canary \
  -o jsonpath='{.status.parents[*].conditions[?(@.type=="Accepted")].status}{"\n"}'

Expected: True True – Linkerd and Cilium have accepted the route.

Challenge

Build a route with which testers always get v2 via x-canary: true, while everyone else is distributed 90/10. Prove both.

Hint: A route may have several rules; the one with matches wins.

Solution

cat > route-challenge.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-canary, namespace: tag09}
spec:
  parentRefs:
  - {group: "", kind: Service, name: notes-api, port: 80}
  rules:
  - matches:
    - headers: [{name: x-canary, value: "true"}]
    backendRefs:
    - {name: notes-api-v2, port: 80}
  - backendRefs:
    - {name: notes-api, port: 80, weight: 90}
    - {name: notes-api-v2, port: 80, weight: 10}
EOF
kubectl apply -f route-challenge.yaml
sleep 60
kubectl -n tag09 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
for i in 1 2 3 4 5; do
  kubectl -n tag09 exec deploy/last -c last -- \
    curl -s -H 'x-canary: true' http://notes-api/version
done | sort | uniq -c

Expected (numbers vary):

     79 0.5.2 200
     12 0.7.1 200
      5 0.7.1

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag09

Cilium’s EndpointSlices disappear along with the namespace. If you deleted routes with a Service parent in other namespaces, check there with kubectl get endpointslices -A | grep 192.192.192.192.

Interview Note

  • How does Linkerd do a canary? With a Gateway API HTTPRoute whose parent is the Service; backendRefs with weights distribute in the client proxy. Older guides use the SMI resource TrafficSplit – for anything new, you use HTTPRoute.
  • Why does a client without a proxy not see the route? The caller’s outbound proxy implements the route. Without a proxy, only the Service applies.
  • How do you roll back? Set the weight of the new version to 0 or delete the route – takes effect immediately, without a restart, as long as the old version is still running.

Enterprise Note

  • By hand, this is a learning tool. In production, Flagger or Argo Rollouts (with the Gateway API plugin) shifts the weights of the same HTTPRoute, measures the success rate from Prometheus, and rolls back automatically on deviation.
  • A GitOps tool must then not overwrite the weights – ignore the fields or leave the route entirely to the rollout tool.
  • Only one GAMMA implementation per Service: if Cilium with Gateway API runs alongside, it also accepts routes with a Service parent – check status.parents of every route.
  • Header routing requires header propagation through all services in the chain – as with tracing.
  • Canary weights are request shares, not users: for “a user stays on one version”, you need headers or cookies.