Day 5: Canary and Blue-Green
Goal: You shift traffic between two versions in a controlled way – by weight, by header and with a switch – and measure what really arrives. · Duration: approx. 60 min · Prerequisite: Day 4
Briefing
- Several
backendRefsin one rule withweight= weighted distribution. This is a Core feature, not an annotation likecanary-weightin ingress-nginx. - Weights are relative: 90/10, 9/1 and 900/100 mean the same thing.
weight: 0= no traffic. - Header canary = a separate rule with
matches.headers. More specific rules win. - Blue-green = two Deployments, the default rule points to exactly one. Switching is a single patch.
- Don’t trust any percentage without measuring: we count 200 responses.
┌─ weight 90 ─> shop-v1 (stable / blue)
HTTPRoute shop┤
└─ weight 10 ─> shop-v2 (canary / green)
x-canary: always ──────────> shop-v2
Drill 1: The Measuring Device
echo-basic returns the service name in the JSON. The
script counts it over N requests.
mkdir -p ~/gw-lab/tag05 && cd ~/gw-lab/tag05
export GW_IP=$(gw-ip.sh)
cat > measure.sh <<'EOF'
#!/usr/bin/env bash
# Sends N requests to the shop and counts the responses per backend.
# Usage: measure.sh [N] [additional curl arguments ...]
set -uo pipefail
n="${1:-200}"; shift || true
GW_IP="${GW_IP:-$(gw-ip.sh)}"
CA="${CA:-$HOME/gw-lab/tag04/lab-ca.crt}"
for _ in $(seq "$n"); do
body=$(curl -s --max-time 2 --cacert "$CA" \
--resolve "shop.gw.localtest.me:443:$GW_IP" "$@" \
https://shop.gw.localtest.me/)
jq -er .service <<<"$body" 2>/dev/null || echo FEHLER
done | sort | uniq -c
EOF
chmod +x measure.sh
./measure.sh 20Expected:
20 shop-v1
Anything that is not JSON with service (timeout, 503
from Envoy) counts as FEHLER.
Save your progress? Sign up for free
Drill 2: 90/10 by Weight
cat > canary-weights.yaml <<'EOF'
# 90/10 canary: 10% of requests go to shop-v2
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-shop
hostnames:
- shop.gw.localtest.me
rules:
- backendRefs:
- name: shop-v1
port: 80
weight: 90
- name: shop-v2
port: 80
weight: 10
EOF
kubectl apply -f canary-weights.yaml
sleep 3
./measure.sh 200Expected (numbers fluctuate):
httproute.gateway.networking.k8s.io/shop configured
179 shop-v1
21 shop-v2
21 instead of 20 is not an error: Envoy rolls the dice per request. With 200 requests, ±6 is normal. For tight gates (e.g. 1%) you need many more requests or real metrics (Day 11).
Tip: Weight ≠ users. Without session affinity, the same customer can jump between v1 and v2. For consistent behavior per customer: header canary (Drill 3).
Save your progress? Sign up for free
Drill 3: Header Canary for Testers
Testers with x-canary: always always see v2, everyone
else sees v1:
cat > canary-header.yaml <<'EOF'
# Header canary: testers with x-canary: always always land on v2
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-shop
hostnames:
- shop.gw.localtest.me
rules:
- matches:
- headers:
- name: x-canary
value: always
backendRefs:
- name: shop-v2
port: 80
- backendRefs:
- name: shop-v1
port: 80
weight: 100
- name: shop-v2
port: 80
weight: 0
EOF
kubectl apply -f canary-header.yaml
sleep 3
./measure.sh 50
./measure.sh 50 -H 'x-canary: always'
./measure.sh 20 -H 'X-Canary: always'
./measure.sh 20 -H 'x-canary: ALWAYS'Expected:
httproute.gateway.networking.k8s.io/shop configured
50 shop-v1
50 shop-v2
20 shop-v2
20 shop-v1
Header names are case-insensitive,
values are not. weight: 0 keeps shop-v2
ready in the default rule – we’ll turn it up there in a moment.
Save your progress? Sign up for free
Drill 4: Gradual Rollout by Script
The script turns up the weight of shop-v2 in stages, measures after
each stage and aborts on errors. It patches rules[0] – so
back to the pure weight route.
cat > rollout.sh <<'EOF'
#!/usr/bin/env bash
# Gradual canary: increase the weight of shop-v2, measure after every step.
# On errors, immediately back to 100% shop-v1.
# Usage: rollout.sh [steps ...] Default: 10 25 50 100
set -euo pipefail
if [ $# -gt 0 ]; then steps=("$@"); else steps=(10 25 50 100); fi
N="${N:-200}"
MAX_ERR="${MAX_ERR:-0}"
set_weights() { # $1 = weight v1, $2 = weight v2
kubectl -n shop patch httproute shop --type=json -p "[
{\"op\":\"replace\",\"path\":\"/spec/rules/0/backendRefs/0/weight\",\"value\":$1},
{\"op\":\"replace\",\"path\":\"/spec/rules/0/backendRefs/1/weight\",\"value\":$2}
]" >/dev/null
}
for w in "${steps[@]}"; do
set_weights $((100 - w)) "$w"
sleep "${SETTLE:-3}"
result=$("$(dirname "$0")/measure.sh" "$N")
v2=$(awk '$2=="shop-v2"{print $1}' <<<"$result"); v2=${v2:-0}
err=$(awk '$2=="FEHLER"{print $1}' <<<"$result"); err=${err:-0}
printf 'Schritt %3s%%: shop-v2=%3s/%s Fehler=%s\n' "$w" "$v2" "$N" "$err"
if [ "$err" -gt "$MAX_ERR" ]; then
set_weights 100 0
echo "ROLLBACK: $err Fehler > $MAX_ERR – zurück auf 100 % shop-v1"
exit 1
fi
done
echo "Rollout fertig: 100 % shop-v2"
EOF
chmod +x rollout.sh
kubectl apply -f canary-weights.yaml
./rollout.sh
kubectl -n shop get httproute shop \
-o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}={.weight} {end}{"\n"}'Expected (numbers fluctuate):
httproute.gateway.networking.k8s.io/shop configured
Schritt 10%: shop-v2= 20/200 Fehler=0
Schritt 25%: shop-v2= 50/200 Fehler=0
Schritt 50%: shop-v2=101/200 Fehler=0
Schritt 100%: shop-v2=200/200 Fehler=0
Rollout fertig: 100 % shop-v2
shop-v1=0 shop-v2=100
At its core, this is what Argo Rollouts or Flagger do – only with Prometheus queries instead of a counting loop and with pauses between the stages.
Save your progress? Sign up for free
Drill 5: Rollback When v2 Is Broken
Simulate a broken v2: no more pods behind the Service. Envoy responds with 503 for that share.
kubectl apply -f canary-weights.yaml
kubectl -n shop scale deploy shop-v2 --replicas=0
sleep 5
./rollout.sh 10 50 100; echo "Exit: $?"
kubectl -n shop get httproute shop \
-o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}={.weight} {end}{"\n"}'
./measure.sh 20
kubectl -n shop scale deploy shop-v2 --replicas=1Expected (numbers fluctuate):
httproute.gateway.networking.k8s.io/shop configured
deployment.apps/shop-v2 scaled
Schritt 10%: shop-v2= 0/200 Fehler=22
ROLLBACK: 22 Fehler > 0 – zurück auf 100 % shop-v1
Exit: 1
shop-v1=100 shop-v2=0
20 shop-v1
deployment.apps/shop-v2 scaled
The rollback is the same mechanism as the rollout: a patch of the weights. No redeploy, no new pod generation, effective within seconds.
Warning: Weighted distribution knows nothing about health. A backend without endpoints still gets its share – and returns 503. Always gate on the error rate, not just on “pods Ready”.
Save your progress? Sign up for free
Drill 6: Blue-Green With Preview and Switch
Blue (shop-v1) is live. Green (shop-v2) is only reachable with
x-env: green – for acceptance testing before switching.
cat > blue-green.yaml <<'EOF'
# Blue-green: blue (shop-v1) gets everything, green (shop-v2) only with x-env: green
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-shop
hostnames:
- shop.gw.localtest.me
rules:
- matches:
- headers:
- name: x-env
value: green
backendRefs:
- name: shop-v2
port: 80
- backendRefs:
- name: shop-v1
port: 80
EOF
cat > switch.sh <<'EOF'
#!/usr/bin/env bash
# Atomically switches the default rule of the route shop to blue (v1) or green (v2).
# Usage: switch.sh blue|green
set -euo pipefail
case "${1:-}" in
blue) svc=shop-v1 ;;
green) svc=shop-v2 ;;
*) echo "Aufruf: $0 blue|green" >&2; exit 2 ;;
esac
kubectl -n shop patch httproute shop --type=json -p "[
{\"op\":\"replace\",\"path\":\"/spec/rules/1/backendRefs/0/name\",\"value\":\"$svc\"}
]"
echo "live: $1 ($svc)"
EOF
chmod +x switch.sh
kubectl -n shop rollout status deploy/shop-v2 --timeout=60s
kubectl apply -f blue-green.yaml
sleep 3
./measure.sh 20
./measure.sh 20 -H 'x-env: green'Expected:
deployment "shop-v2" successfully rolled out
httproute.gateway.networking.k8s.io/shop configured
20 shop-v1
20 shop-v2
Now switch – while 300 requests are running in the background:
./measure.sh 300 > waehrend.txt &
sleep 0.5
./switch.sh green
wait
cat waehrend.txt
./measure.sh 20Expected (distribution fluctuates):
httproute.gateway.networking.k8s.io/shop patched
live: green (shop-v2)
76 shop-v1
224 shop-v2
20 shop-v2
Not a single FEHLER: Envoy swaps the route configuration
without dropping connections. Rollback is
./switch.sh blue.
Save your progress? Sign up for free
Drill 7: Argo Rollouts and Flagger (Outlook)
Both tools patch exactly the weights you just turned by hand – only
driven by metrics. Argo Rollouts needs the plugin
argoproj-labs/gatewayAPI for this, Flagger knows the
Gateway API as the provider gatewayapi:v1.
# Argo Rollouts: excerpt of a Rollout (plugin argoproj-labs/gatewayAPI)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: shop
namespace: shop
spec:
strategy:
canary:
stableService: shop-stable
canaryService: shop-canary
trafficRouting:
plugins:
argoproj-labs/gatewayAPI:
httpRoute: shop
namespace: shop
steps:
- setWeight: 10
- pause: {duration: 5m}
- setWeight: 50
- pause: {duration: 10m}(shortened, may differ depending on the version)
| Tool | Gateway API integration | Gate |
|---|---|---|
| Argo Rollouts | Plugin argoproj-labs/gatewayAPI |
AnalysisTemplate (Prometheus and others) |
| Flagger | Canary with provider: gatewayapi:v1 |
MetricTemplate, webhooks |
| Your own script | kubectl patch (Drill 4) |
whatever you measure |
Both manage the route themselves: anyone who patches by hand in parallel is fighting the controller.
Save your progress? Sign up for free
Checkpoint
cd ~/gw-lab/tag05
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
export GW_IP=$(gw-ip.sh)
check "measure.sh und rollout.sh ausführbar" '[ -x measure.sh ] && [ -x rollout.sh ]'
check "shop-v2 hat Endpunkte" \
'[ -n "$(kubectl -n shop get endpointslice -l kubernetes.io/service-name=shop-v2 \
-o jsonpath="{.items[*].endpoints[*].addresses[0]}")" ]'
check "Route shop Accepted" \
'conds.sh httproute shop shop | grep -q "Accepted=True"'
check "Vorschau x-env: green -> shop-v2" \
'./measure.sh 5 -H "x-env: green" | grep -q "5 shop-v2"'
check "keine Fehler in 50 Requests" '! ./measure.sh 50 | grep -q FEHLER'Expected:
✔ measure.sh und rollout.sh ausführbar
✔ shop-v2 hat Endpunkte
✔ Route shop Accepted
✔ Vorschau x-env: green -> shop-v2
✔ keine Fehler in 50 Requests
Challenge
The QA team (header x-team: qa) should always see v2,
everyone else gets 95/5. Prove the distribution with 200 requests each
and then restore the state of Day 4 (100% shop-v1).
Hint: Two rules – the header rule is more specific and wins.
Solution
cat > qa-canary.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-shop
hostnames:
- shop.gw.localtest.me
rules:
- matches:
- headers:
- name: x-team
value: qa
backendRefs:
- name: shop-v2
port: 80
- backendRefs:
- name: shop-v1
port: 80
weight: 95
- name: shop-v2
port: 80
weight: 5
EOF
kubectl apply -f qa-canary.yaml
sleep 3
./measure.sh 200
./measure.sh 200 -H 'x-team: qa'
kubectl apply -f ~/gw-lab/tag04/routes-https.yamlExpected (numbers fluctuate):
httproute.gateway.networking.k8s.io/shop configured
191 shop-v1
9 shop-v2
200 shop-v2
httproute.gateway.networking.k8s.io/shop configured
httproute.gateway.networking.k8s.io/api unchanged
Save your progress? Sign up for free
Enterprise Note
- Weights by hand only in an emergency. Day to day, Argo Rollouts or Flagger controls the route, gated on error rate and latency from Prometheus (Day 11).
- It must be clear who owns the route: otherwise GitOps (Argo CD/Flux)
and the rollout controller modify the same object – set
ignoreDifferencesforbackendRefs[].weight. - Canary by weight needs volume: with little traffic, test header- or cookie-based instead and then switch.
- Database migrations must tolerate v1 and v2 at the same time (expand/contract), otherwise the fast rollback only seems fast.
- Blue-green costs double capacity during the switchover phase – account for it in the capacity plan and in ResourceQuotas.