Day 12: Blue/Green and Canary
Goal: You switch notes-api from one version to another via the Service selector and send a small share of the requests to a new version. · Duration: approx. 60 min · Prerequisite: Day 11
Briefing
CKAD: “Use Kubernetes primitives to implement common deployment strategies (e.g. blue/green or canary)” (Application Deployment, 20 %).
Blue/Green Canary
Service selector version=blue Service selector app=notes-c
| |
+--> notes-blue (1.0) x2 +--> notes-stable (1.0) x4 ~80 %
notes-green (1.1) x2 waiting +--> notes-canary (1.1) x1 ~20 %
switch = change selector ratio = replica count
- Tools: two Deployments, one Service, labels. No extra product needed.
- Images from Day 11:
notes-api:1.0andnotes-api:1.1;/versionshows who answers.
mkdir -p ~/ckad/tag12 && cd ~/ckad/tag12
k create ns tag12
k config set-context --current --namespace=tag12Drill 1: Deploy Blue and Green
Skeleton with k create deploy notes-blue … $do, then add
labels and env:
cat > blue.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes-blue
spec:
replicas: 2
selector:
matchLabels:
app: notes
version: blue
template:
metadata:
labels:
app: notes
version: blue
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
EOF
sed -e 's/blue/green/g' -e 's/notes-api:1.0/notes-api:1.1/' blue.yaml > green.yaml
k apply -f blue.yaml -f green.yaml
k rollout status deploy/notes-blue
k rollout status deploy/notes-green
k get pods -L versionExpected (shortened):
deployment.apps/notes-blue created
deployment.apps/notes-green created
deployment "notes-blue" successfully rolled out
deployment "notes-green" successfully rolled out
NAME READY STATUS RESTARTS AGE VERSION
notes-blue-556bc89579-mrqmw 1/1 Running 0 3s blue
notes-blue-556bc89579-xsfwn 1/1 Running 0 3s blue
notes-green-78dbc96979-l29ks 1/1 Running 0 3s green
notes-green-78dbc96979-sr4bf 1/1 Running 0 3s green
lab-key-123 is purely a lab value. Green is already
running but gets no traffic yet – you can test the new version calmly
beforehand.
Save your progress? Sign up for free
Drill 2: The Service points to Blue
k expose takes over the Deployment’s selector –
i.e. app=notes,version=blue:
k expose deploy notes-blue --name=notes --port=80 --target-port=8080
k get svc notes -o jsonpath='{.spec.selector}{"\n"}'
IP=$(k get svc notes -o jsonpath='{.spec.clusterIP}')
for i in $(seq 20); do curl -s $IP/version; done | sort | uniq -cExpected:
service/notes exposed
{"app":"notes","version":"blue"}
20 1.0
The loop is a single command: 20 requests, counted by response. From
cp1 the ClusterIP is directly reachable. If the output is
empty right after expose, kube-proxy has not set up the
rules yet – wait a second and repeat.
Save your progress? Sign up for free
Drill 3: Switch to Green
k patch svc notes -p '{"spec":{"selector":{"version":"green"}}}'
for i in $(seq 20); do curl -s $IP/version; done | sort | uniq -cExpected:
service/notes patched
20 1.1
One command, and after about a second (kube-proxy updates its rules)
all requests go to Green. patch merges:
app=notes stays, only version changes.
Save your progress? Sign up for free
Drill 4: Back to Blue
Something wrong with 1.1? The rollback is just as fast – Blue is still running:
k set selector svc notes 'app=notes,version=blue'
k get svc notes -o jsonpath='{.spec.selector}{"\n"}'
for i in $(seq 20); do curl -s $IP/version; done | sort | uniq -cExpected:
service/notes selector updated
{"app":"notes","version":"blue"}
20 1.0
k set selector replaces the entire selector – so specify
both labels. The price of Blue/Green: both versions occupy full
resources at the same time.
Save your progress? Sign up for free
Drill 5: Canary at 4:1
For a canary, the Service selects both Deployments via a shared label. The ratio of the replicas determines the share:
sed -e 's/notes-blue/notes-stable/' -e 's/app: notes$/app: notes-c/' \
-e 's/version: blue/track: stable/' -e 's/replicas: 2/replicas: 4/' blue.yaml \
> stable.yaml
sed -e 's/stable/canary/g' -e 's/replicas: 4/replicas: 1/' \
-e 's/notes-api:1.0/notes-api:1.1/' stable.yaml > canary.yaml
grep -A3 'labels:' canary.yaml
k apply -f stable.yaml -f canary.yaml
k create service clusterip notes-c --tcp=80:8080
k get svc notes-c -o jsonpath='{.spec.selector}{"\n"}'Expected:
labels:
app: notes-c
track: canary
spec:
deployment.apps/notes-stable created
deployment.apps/notes-canary created
service/notes-c created
{"app":"notes-c"}
k create service sets the selector
app=<name> – here exactly app=notes-c,
the shared label.
k rollout status deploy/notes-stable
k get pods -l app=notes-c -L track
CIP=$(k get svc notes-c -o jsonpath='{.spec.clusterIP}')
for i in $(seq 20); do curl -s $CIP/version; done | sort | uniq -cExpected (distribution varies):
deployment "notes-stable" successfully rolled out
NAME READY STATUS RESTARTS AGE TRACK
notes-canary-7f7f7f88f9-9mxnj 1/1 Running 0 4s canary
notes-stable-d4746df97-4l7f6 1/1 Running 0 4s stable
notes-stable-d4746df97-b5l9k 1/1 Running 0 4s stable
notes-stable-d4746df97-nrmrj 1/1 Running 0 4s stable
notes-stable-d4746df97-qz9ld 1/1 Running 0 4s stable
18 1.0
2 1.1
kube-proxy distributes randomly, not round-robin. With 20 requests, 1
to 7 hits on 1.1 are normal; with seq 200 the ratio becomes
clearer (152 to 48 in testing).
Save your progress? Sign up for free
Drill 6: Expand or abort the canary
The canary is doing well – go to 50 %:
k scale deploy notes-stable --replicas=2
k scale deploy notes-canary --replicas=2
for i in $(seq 20); do curl -s $CIP/version; done | sort | uniq -cExpected (distribution varies):
deployment.apps/notes-stable scaled
deployment.apps/notes-canary scaled
12 1.0
8 1.1
And the rollback, in case 1.1 throws errors – immediately, without restarting the stable pods:
k scale deploy notes-canary --replicas=0
k scale deploy notes-stable --replicas=4
for i in $(seq 20); do curl -s $CIP/version; done | sort | uniq -cExpected:
deployment.apps/notes-canary scaled
deployment.apps/notes-stable scaled
20 1.0
For the complete switchover, you finally set
notes-stable to 1.1 with k set image (rolling
update, Day 11) and scale the canary to 0.
Save your progress? Sign up for free
Drill 7: Who is behind the Service?
k get endpointslices -l kubernetes.io/service-name=notes-c
k get pods -l app=notes-c -o wide | awk '{print $1, $6}'Expected (shortened):
NAME ADDRESSTYPE PORTS ENDPOINTS
notes-c-lfm76 IPv4 8080 192.168.189.80,192.168.189.108,... + 1 more...
NAME IP
notes-stable-d4746df97-b5l9k 192.168.189.80
notes-stable-d4746df97-m5bmf 192.168.235.158
notes-stable-d4746df97-qz9ld 192.168.189.108
notes-stable-d4746df97-rg5mz 192.168.235.184
The EndpointSlice shows the pod IPs the Service is currently distributing to. If something is off here, the selector and the pod labels do not match – the most common mistake with both strategies.
Save your progress? Sign up for free
Exercise: Speed Drills
- Switch the Service
notesto Green. Target time: 20 s - Start a canary again with 1 of 5 pods on 1.1. Target time: 20 s
- Show the labels of all pods with
app=notes. Target time: 15 s
Solution
k set selector svc notes 'app=notes,version=green'
k scale deploy notes-canary --replicas=1
k get pods -l app=notes --show-labelsExpected (shortened):
service/notes selector updated
deployment.apps/notes-canary scaled
NAME ... LABELS
notes-blue-556bc89579-mrqmw ... app=notes,pod-template-hash=556bc89579,version=blue
notes-blue-556bc89579-xsfwn ... app=notes,pod-template-hash=556bc89579,version=blue
notes-green-78dbc96979-l29ks ... app=notes,pod-template-hash=78dbc96979,version=green
notes-green-78dbc96979-sr4bf ... app=notes,pod-template-hash=78dbc96979,version=green
Save your progress? Sign up for free
Checkpoint
k get svc notes -o jsonpath='{.spec.selector.version}{"\n"}'Expected: green
k get deploy notes-stable notes-canaryExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
notes-stable 4/4 4 4 39s
notes-canary 1/1 1 1 39s
Challenge
Make the Blue/Green Service notes reachable via Traefik
under the host notes.tag12.lab and switch back to Blue in a
way that is visible from outside.
Hint: Traefik is the default IngressClass and listens on NodePort 30080.
Solution
k create ingress notes --rule='notes.tag12.lab/*=notes:80'
curl -s -H 'Host: notes.tag12.lab' http://172.16.0.126:30080/version
k set selector svc notes 'app=notes,version=blue'
curl -s -H 'Host: notes.tag12.lab' http://172.16.0.126:30080/versionExpected:
ingress.networking.k8s.io/notes created
1.1
service/notes selector updated
1.0
The Ingress does not change – switching still happens at the Service.
Traefik picks up the new endpoints after one or two seconds; if you
still get 1.1 right afterwards, simply repeat the last
curl.
Clean up:
k config set-context --current --namespace=default
k delete ns tag12Save your progress? Sign up for free
Exam Tip
- Blue/Green = change the Service’s selector
(
k set selector,k patch svc,k edit svc). Canary = shared label in the Service selector, share viareplicas. - Check whether pod labels and selector match:
k get pods --show-labels,k get endpointslices -l kubernetes.io/service-name=<svc>. - Change labels in the template, not just in the Deployment’s
metadata– and a Deployment’sselector.matchLabelsis immutable after creation.
Enterprise Note
- A canary via replica count is coarse (1 of 5 = 20 %). Finer weights
come from the ingress controller: Traefik with a
TraefikServiceof typeweighted, Gateway API withbackendRefs[].weight. - Argo Rollouts and Flagger automate canary stages and abort on their own when metrics are bad.
- With Blue/Green, keep database schemas backward compatible – otherwise there is no way back.
- Consider sessions (sticky sessions, caches) when switching.