Skip to content
Hands-on IT IT Drills

Day 5: Canary and Blue-Green

Goal: You shift traffic between two versions in a controlled way – by weight, by header and with a switch – and measure what really arrives. · Duration: approx. 60 min · Prerequisite: Day 4

Briefing

  • Several backendRefs in one rule with weight = weighted distribution. This is a Core feature, not an annotation like canary-weight in ingress-nginx.
  • Weights are relative: 90/10, 9/1 and 900/100 mean the same thing. weight: 0 = no traffic.
  • Header canary = a separate rule with matches.headers. More specific rules win.
  • Blue-green = two Deployments, the default rule points to exactly one. Switching is a single patch.
  • Don’t trust any percentage without measuring: we count 200 responses.
              ┌─ weight 90 ─> shop-v1 (stable / blue)
HTTPRoute shop┤
              └─ weight 10 ─> shop-v2 (canary / green)
x-canary: always ──────────> shop-v2

Drill 1: The Measuring Device

echo-basic returns the service name in the JSON. The script counts it over N requests.

mkdir -p ~/gw-lab/tag05 && cd ~/gw-lab/tag05
export GW_IP=$(gw-ip.sh)
cat > measure.sh <<'EOF'
#!/usr/bin/env bash
# Sends N requests to the shop and counts the responses per backend.
# Usage: measure.sh [N] [additional curl arguments ...]
set -uo pipefail
n="${1:-200}"; shift || true
GW_IP="${GW_IP:-$(gw-ip.sh)}"
CA="${CA:-$HOME/gw-lab/tag04/lab-ca.crt}"
for _ in $(seq "$n"); do
  body=$(curl -s --max-time 2 --cacert "$CA" \
    --resolve "shop.gw.localtest.me:443:$GW_IP" "$@" \
    https://shop.gw.localtest.me/)
  jq -er .service <<<"$body" 2>/dev/null || echo FEHLER
done | sort | uniq -c
EOF
chmod +x measure.sh
./measure.sh 20

Expected:

  20 shop-v1

Anything that is not JSON with service (timeout, 503 from Envoy) counts as FEHLER.

Save your progress? Sign up for free

Drill 2: 90/10 by Weight

cat > canary-weights.yaml <<'EOF'
# 90/10 canary: 10% of requests go to shop-v2
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-shop
  hostnames:
    - shop.gw.localtest.me
  rules:
    - backendRefs:
        - name: shop-v1
          port: 80
          weight: 90
        - name: shop-v2
          port: 80
          weight: 10
EOF
kubectl apply -f canary-weights.yaml
sleep 3
./measure.sh 200

Expected (numbers fluctuate):

httproute.gateway.networking.k8s.io/shop configured
 179 shop-v1
  21 shop-v2

21 instead of 20 is not an error: Envoy rolls the dice per request. With 200 requests, ±6 is normal. For tight gates (e.g. 1%) you need many more requests or real metrics (Day 11).

Tip: Weight ≠ users. Without session affinity, the same customer can jump between v1 and v2. For consistent behavior per customer: header canary (Drill 3).

Save your progress? Sign up for free

Drill 3: Header Canary for Testers

Testers with x-canary: always always see v2, everyone else sees v1:

cat > canary-header.yaml <<'EOF'
# Header canary: testers with x-canary: always always land on v2
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-shop
  hostnames:
    - shop.gw.localtest.me
  rules:
    - matches:
        - headers:
            - name: x-canary
              value: always
      backendRefs:
        - name: shop-v2
          port: 80
    - backendRefs:
        - name: shop-v1
          port: 80
          weight: 100
        - name: shop-v2
          port: 80
          weight: 0
EOF
kubectl apply -f canary-header.yaml
sleep 3
./measure.sh 50
./measure.sh 50 -H 'x-canary: always'
./measure.sh 20 -H 'X-Canary: always'
./measure.sh 20 -H 'x-canary: ALWAYS'

Expected:

httproute.gateway.networking.k8s.io/shop configured
  50 shop-v1
  50 shop-v2
  20 shop-v2
  20 shop-v1

Header names are case-insensitive, values are not. weight: 0 keeps shop-v2 ready in the default rule – we’ll turn it up there in a moment.

Save your progress? Sign up for free

Drill 4: Gradual Rollout by Script

The script turns up the weight of shop-v2 in stages, measures after each stage and aborts on errors. It patches rules[0] – so back to the pure weight route.

cat > rollout.sh <<'EOF'
#!/usr/bin/env bash
# Gradual canary: increase the weight of shop-v2, measure after every step.
# On errors, immediately back to 100% shop-v1.
# Usage: rollout.sh [steps ...]   Default: 10 25 50 100
set -euo pipefail
if [ $# -gt 0 ]; then steps=("$@"); else steps=(10 25 50 100); fi
N="${N:-200}"
MAX_ERR="${MAX_ERR:-0}"

set_weights() {  # $1 = weight v1, $2 = weight v2
  kubectl -n shop patch httproute shop --type=json -p "[
    {\"op\":\"replace\",\"path\":\"/spec/rules/0/backendRefs/0/weight\",\"value\":$1},
    {\"op\":\"replace\",\"path\":\"/spec/rules/0/backendRefs/1/weight\",\"value\":$2}
  ]" >/dev/null
}

for w in "${steps[@]}"; do
  set_weights $((100 - w)) "$w"
  sleep "${SETTLE:-3}"
  result=$("$(dirname "$0")/measure.sh" "$N")
  v2=$(awk '$2=="shop-v2"{print $1}' <<<"$result"); v2=${v2:-0}
  err=$(awk '$2=="FEHLER"{print $1}' <<<"$result"); err=${err:-0}
  printf 'Schritt %3s%%: shop-v2=%3s/%s Fehler=%s\n' "$w" "$v2" "$N" "$err"
  if [ "$err" -gt "$MAX_ERR" ]; then
    set_weights 100 0
    echo "ROLLBACK: $err Fehler > $MAX_ERR – zurück auf 100 % shop-v1"
    exit 1
  fi
done
echo "Rollout fertig: 100 % shop-v2"
EOF
chmod +x rollout.sh
kubectl apply -f canary-weights.yaml
./rollout.sh
kubectl -n shop get httproute shop \
  -o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}={.weight} {end}{"\n"}'

Expected (numbers fluctuate):

httproute.gateway.networking.k8s.io/shop configured
Schritt  10%: shop-v2= 20/200 Fehler=0
Schritt  25%: shop-v2= 50/200 Fehler=0
Schritt  50%: shop-v2=101/200 Fehler=0
Schritt 100%: shop-v2=200/200 Fehler=0
Rollout fertig: 100 % shop-v2
shop-v1=0 shop-v2=100

At its core, this is what Argo Rollouts or Flagger do – only with Prometheus queries instead of a counting loop and with pauses between the stages.

Save your progress? Sign up for free

Drill 5: Rollback When v2 Is Broken

Simulate a broken v2: no more pods behind the Service. Envoy responds with 503 for that share.

kubectl apply -f canary-weights.yaml
kubectl -n shop scale deploy shop-v2 --replicas=0
sleep 5
./rollout.sh 10 50 100; echo "Exit: $?"
kubectl -n shop get httproute shop \
  -o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}={.weight} {end}{"\n"}'
./measure.sh 20
kubectl -n shop scale deploy shop-v2 --replicas=1

Expected (numbers fluctuate):

httproute.gateway.networking.k8s.io/shop configured
deployment.apps/shop-v2 scaled
Schritt  10%: shop-v2=  0/200 Fehler=22
ROLLBACK: 22 Fehler > 0 – zurück auf 100 % shop-v1
Exit: 1
shop-v1=100 shop-v2=0
  20 shop-v1
deployment.apps/shop-v2 scaled

The rollback is the same mechanism as the rollout: a patch of the weights. No redeploy, no new pod generation, effective within seconds.

Warning: Weighted distribution knows nothing about health. A backend without endpoints still gets its share – and returns 503. Always gate on the error rate, not just on “pods Ready”.

Save your progress? Sign up for free

Drill 6: Blue-Green With Preview and Switch

Blue (shop-v1) is live. Green (shop-v2) is only reachable with x-env: green – for acceptance testing before switching.

cat > blue-green.yaml <<'EOF'
# Blue-green: blue (shop-v1) gets everything, green (shop-v2) only with x-env: green
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-shop
  hostnames:
    - shop.gw.localtest.me
  rules:
    - matches:
        - headers:
            - name: x-env
              value: green
      backendRefs:
        - name: shop-v2
          port: 80
    - backendRefs:
        - name: shop-v1
          port: 80
EOF
cat > switch.sh <<'EOF'
#!/usr/bin/env bash
# Atomically switches the default rule of the route shop to blue (v1) or green (v2).
# Usage: switch.sh blue|green
set -euo pipefail
case "${1:-}" in
  blue)  svc=shop-v1 ;;
  green) svc=shop-v2 ;;
  *) echo "Aufruf: $0 blue|green" >&2; exit 2 ;;
esac
kubectl -n shop patch httproute shop --type=json -p "[
  {\"op\":\"replace\",\"path\":\"/spec/rules/1/backendRefs/0/name\",\"value\":\"$svc\"}
]"
echo "live: $1 ($svc)"
EOF
chmod +x switch.sh
kubectl -n shop rollout status deploy/shop-v2 --timeout=60s
kubectl apply -f blue-green.yaml
sleep 3
./measure.sh 20
./measure.sh 20 -H 'x-env: green'

Expected:

deployment "shop-v2" successfully rolled out
httproute.gateway.networking.k8s.io/shop configured
  20 shop-v1
  20 shop-v2

Now switch – while 300 requests are running in the background:

./measure.sh 300 > waehrend.txt &
sleep 0.5
./switch.sh green
wait
cat waehrend.txt
./measure.sh 20

Expected (distribution fluctuates):

httproute.gateway.networking.k8s.io/shop patched
live: green (shop-v2)
  76 shop-v1
 224 shop-v2
  20 shop-v2

Not a single FEHLER: Envoy swaps the route configuration without dropping connections. Rollback is ./switch.sh blue.

Save your progress? Sign up for free

Drill 7: Argo Rollouts and Flagger (Outlook)

Both tools patch exactly the weights you just turned by hand – only driven by metrics. Argo Rollouts needs the plugin argoproj-labs/gatewayAPI for this, Flagger knows the Gateway API as the provider gatewayapi:v1.

# Argo Rollouts: excerpt of a Rollout (plugin argoproj-labs/gatewayAPI)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: shop
  namespace: shop
spec:
  strategy:
    canary:
      stableService: shop-stable
      canaryService: shop-canary
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: shop
            namespace: shop
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - setWeight: 50
        - pause: {duration: 10m}

(shortened, may differ depending on the version)

Tool Gateway API integration Gate
Argo Rollouts Plugin argoproj-labs/gatewayAPI AnalysisTemplate (Prometheus and others)
Flagger Canary with provider: gatewayapi:v1 MetricTemplate, webhooks
Your own script kubectl patch (Drill 4) whatever you measure

Both manage the route themselves: anyone who patches by hand in parallel is fighting the controller.

Save your progress? Sign up for free

Checkpoint

cd ~/gw-lab/tag05
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
export GW_IP=$(gw-ip.sh)
check "measure.sh und rollout.sh ausführbar" '[ -x measure.sh ] && [ -x rollout.sh ]'
check "shop-v2 hat Endpunkte" \
  '[ -n "$(kubectl -n shop get endpointslice -l kubernetes.io/service-name=shop-v2 \
     -o jsonpath="{.items[*].endpoints[*].addresses[0]}")" ]'
check "Route shop Accepted" \
  'conds.sh httproute shop shop | grep -q "Accepted=True"'
check "Vorschau x-env: green -> shop-v2" \
  './measure.sh 5 -H "x-env: green" | grep -q "5 shop-v2"'
check "keine Fehler in 50 Requests" '! ./measure.sh 50 | grep -q FEHLER'

Expected:

✔ measure.sh und rollout.sh ausführbar
✔ shop-v2 hat Endpunkte
✔ Route shop Accepted
✔ Vorschau x-env: green -> shop-v2
✔ keine Fehler in 50 Requests

Challenge

The QA team (header x-team: qa) should always see v2, everyone else gets 95/5. Prove the distribution with 200 requests each and then restore the state of Day 4 (100% shop-v1).

Hint: Two rules – the header rule is more specific and wins.

Solution

cat > qa-canary.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-shop
  hostnames:
    - shop.gw.localtest.me
  rules:
    - matches:
        - headers:
            - name: x-team
              value: qa
      backendRefs:
        - name: shop-v2
          port: 80
    - backendRefs:
        - name: shop-v1
          port: 80
          weight: 95
        - name: shop-v2
          port: 80
          weight: 5
EOF
kubectl apply -f qa-canary.yaml
sleep 3
./measure.sh 200
./measure.sh 200 -H 'x-team: qa'
kubectl apply -f ~/gw-lab/tag04/routes-https.yaml

Expected (numbers fluctuate):

httproute.gateway.networking.k8s.io/shop configured
 191 shop-v1
   9 shop-v2
 200 shop-v2
httproute.gateway.networking.k8s.io/shop configured
httproute.gateway.networking.k8s.io/api unchanged

Save your progress? Sign up for free

Enterprise Note

  • Weights by hand only in an emergency. Day to day, Argo Rollouts or Flagger controls the route, gated on error rate and latency from Prometheus (Day 11).
  • It must be clear who owns the route: otherwise GitOps (Argo CD/Flux) and the rollout controller modify the same object – set ignoreDifferences for backendRefs[].weight.
  • Canary by weight needs volume: with little traffic, test header- or cookie-based instead and then switch.
  • Database migrations must tolerate v1 and v2 at the same time (expand/contract), otherwise the fast rollback only seems fast.
  • Blue-green costs double capacity during the switchover phase – account for it in the capacity plan and in ResourceQuotas.