Skip to content
Hands-on IT IT Drills

Day 12: Blue/Green and Canary

Goal: You switch notes-api from one version to another via the Service selector and send a small share of the requests to a new version. · Duration: approx. 60 min · Prerequisite: Day 11

Briefing

CKAD: “Use Kubernetes primitives to implement common deployment strategies (e.g. blue/green or canary)” (Application Deployment, 20 %).

  Blue/Green                          Canary
  Service selector version=blue       Service selector app=notes-c
     |                                   |
     +--> notes-blue  (1.0) x2           +--> notes-stable (1.0) x4  ~80 %
          notes-green (1.1) x2 waiting   +--> notes-canary (1.1) x1  ~20 %
  switch = change selector            ratio = replica count
  • Tools: two Deployments, one Service, labels. No extra product needed.
  • Images from Day 11: notes-api:1.0 and notes-api:1.1; /version shows who answers.
mkdir -p ~/ckad/tag12 && cd ~/ckad/tag12
k create ns tag12
k config set-context --current --namespace=tag12

Drill 1: Deploy Blue and Green

Skeleton with k create deploy notes-blue … $do, then add labels and env:

cat > blue.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-blue
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
      version: blue
  template:
    metadata:
      labels:
        app: notes
        version: blue
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
EOF
sed -e 's/blue/green/g' -e 's/notes-api:1.0/notes-api:1.1/' blue.yaml > green.yaml
k apply -f blue.yaml -f green.yaml
k rollout status deploy/notes-blue
k rollout status deploy/notes-green
k get pods -L version

Expected (shortened):

deployment.apps/notes-blue created
deployment.apps/notes-green created
deployment "notes-blue" successfully rolled out
deployment "notes-green" successfully rolled out
NAME                           READY   STATUS    RESTARTS   AGE   VERSION
notes-blue-556bc89579-mrqmw    1/1     Running   0          3s    blue
notes-blue-556bc89579-xsfwn    1/1     Running   0          3s    blue
notes-green-78dbc96979-l29ks   1/1     Running   0          3s    green
notes-green-78dbc96979-sr4bf   1/1     Running   0          3s    green

lab-key-123 is purely a lab value. Green is already running but gets no traffic yet – you can test the new version calmly beforehand.

Save your progress? Sign up for free

Drill 2: The Service points to Blue

k expose takes over the Deployment’s selector – i.e. app=notes,version=blue:

k expose deploy notes-blue --name=notes --port=80 --target-port=8080
k get svc notes -o jsonpath='{.spec.selector}{"\n"}'
IP=$(k get svc notes -o jsonpath='{.spec.clusterIP}')
for i in $(seq 20); do curl -s $IP/version; done | sort | uniq -c

Expected:

service/notes exposed
{"app":"notes","version":"blue"}
     20 1.0

The loop is a single command: 20 requests, counted by response. From cp1 the ClusterIP is directly reachable. If the output is empty right after expose, kube-proxy has not set up the rules yet – wait a second and repeat.

Save your progress? Sign up for free

Drill 3: Switch to Green

k patch svc notes -p '{"spec":{"selector":{"version":"green"}}}'
for i in $(seq 20); do curl -s $IP/version; done | sort | uniq -c

Expected:

service/notes patched
     20 1.1

One command, and after about a second (kube-proxy updates its rules) all requests go to Green. patch merges: app=notes stays, only version changes.

Save your progress? Sign up for free

Drill 4: Back to Blue

Something wrong with 1.1? The rollback is just as fast – Blue is still running:

k set selector svc notes 'app=notes,version=blue'
k get svc notes -o jsonpath='{.spec.selector}{"\n"}'
for i in $(seq 20); do curl -s $IP/version; done | sort | uniq -c

Expected:

service/notes selector updated
{"app":"notes","version":"blue"}
     20 1.0

k set selector replaces the entire selector – so specify both labels. The price of Blue/Green: both versions occupy full resources at the same time.

Save your progress? Sign up for free

Drill 5: Canary at 4:1

For a canary, the Service selects both Deployments via a shared label. The ratio of the replicas determines the share:

sed -e 's/notes-blue/notes-stable/' -e 's/app: notes$/app: notes-c/' \
  -e 's/version: blue/track: stable/' -e 's/replicas: 2/replicas: 4/' blue.yaml \
  > stable.yaml
sed -e 's/stable/canary/g' -e 's/replicas: 4/replicas: 1/' \
  -e 's/notes-api:1.0/notes-api:1.1/' stable.yaml > canary.yaml
grep -A3 'labels:' canary.yaml
k apply -f stable.yaml -f canary.yaml
k create service clusterip notes-c --tcp=80:8080
k get svc notes-c -o jsonpath='{.spec.selector}{"\n"}'

Expected:

      labels:
        app: notes-c
        track: canary
    spec:
deployment.apps/notes-stable created
deployment.apps/notes-canary created
service/notes-c created
{"app":"notes-c"}

k create service sets the selector app=<name> – here exactly app=notes-c, the shared label.

k rollout status deploy/notes-stable
k get pods -l app=notes-c -L track
CIP=$(k get svc notes-c -o jsonpath='{.spec.clusterIP}')
for i in $(seq 20); do curl -s $CIP/version; done | sort | uniq -c

Expected (distribution varies):

deployment "notes-stable" successfully rolled out
NAME                            READY   STATUS    RESTARTS   AGE   TRACK
notes-canary-7f7f7f88f9-9mxnj   1/1     Running   0          4s    canary
notes-stable-d4746df97-4l7f6    1/1     Running   0          4s    stable
notes-stable-d4746df97-b5l9k    1/1     Running   0          4s    stable
notes-stable-d4746df97-nrmrj    1/1     Running   0          4s    stable
notes-stable-d4746df97-qz9ld    1/1     Running   0          4s    stable
     18 1.0
      2 1.1

kube-proxy distributes randomly, not round-robin. With 20 requests, 1 to 7 hits on 1.1 are normal; with seq 200 the ratio becomes clearer (152 to 48 in testing).

Save your progress? Sign up for free

Drill 6: Expand or abort the canary

The canary is doing well – go to 50 %:

k scale deploy notes-stable --replicas=2
k scale deploy notes-canary --replicas=2
for i in $(seq 20); do curl -s $CIP/version; done | sort | uniq -c

Expected (distribution varies):

deployment.apps/notes-stable scaled
deployment.apps/notes-canary scaled
     12 1.0
      8 1.1

And the rollback, in case 1.1 throws errors – immediately, without restarting the stable pods:

k scale deploy notes-canary --replicas=0
k scale deploy notes-stable --replicas=4
for i in $(seq 20); do curl -s $CIP/version; done | sort | uniq -c

Expected:

deployment.apps/notes-canary scaled
deployment.apps/notes-stable scaled
     20 1.0

For the complete switchover, you finally set notes-stable to 1.1 with k set image (rolling update, Day 11) and scale the canary to 0.

Save your progress? Sign up for free

Drill 7: Who is behind the Service?

k get endpointslices -l kubernetes.io/service-name=notes-c
k get pods -l app=notes-c -o wide | awk '{print $1, $6}'

Expected (shortened):

NAME            ADDRESSTYPE   PORTS   ENDPOINTS
notes-c-lfm76   IPv4          8080    192.168.189.80,192.168.189.108,... + 1 more...
NAME IP
notes-stable-d4746df97-b5l9k 192.168.189.80
notes-stable-d4746df97-m5bmf 192.168.235.158
notes-stable-d4746df97-qz9ld 192.168.189.108
notes-stable-d4746df97-rg5mz 192.168.235.184

The EndpointSlice shows the pod IPs the Service is currently distributing to. If something is off here, the selector and the pod labels do not match – the most common mistake with both strategies.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Switch the Service notes to Green. Target time: 20 s
  2. Start a canary again with 1 of 5 pods on 1.1. Target time: 20 s
  3. Show the labels of all pods with app=notes. Target time: 15 s

Solution

k set selector svc notes 'app=notes,version=green'
k scale deploy notes-canary --replicas=1
k get pods -l app=notes --show-labels

Expected (shortened):

service/notes selector updated
deployment.apps/notes-canary scaled
NAME                           ...  LABELS
notes-blue-556bc89579-mrqmw    ...  app=notes,pod-template-hash=556bc89579,version=blue
notes-blue-556bc89579-xsfwn    ...  app=notes,pod-template-hash=556bc89579,version=blue
notes-green-78dbc96979-l29ks   ...  app=notes,pod-template-hash=78dbc96979,version=green
notes-green-78dbc96979-sr4bf   ...  app=notes,pod-template-hash=78dbc96979,version=green

Save your progress? Sign up for free

Checkpoint

k get svc notes -o jsonpath='{.spec.selector.version}{"\n"}'

Expected: green

k get deploy notes-stable notes-canary

Expected:

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
notes-stable   4/4     4            4           39s
notes-canary   1/1     1            1           39s

Challenge

Make the Blue/Green Service notes reachable via Traefik under the host notes.tag12.lab and switch back to Blue in a way that is visible from outside.

Hint: Traefik is the default IngressClass and listens on NodePort 30080.

Solution

k create ingress notes --rule='notes.tag12.lab/*=notes:80'
curl -s -H 'Host: notes.tag12.lab' http://172.16.0.126:30080/version
k set selector svc notes 'app=notes,version=blue'
curl -s -H 'Host: notes.tag12.lab' http://172.16.0.126:30080/version

Expected:

ingress.networking.k8s.io/notes created
1.1
service/notes selector updated
1.0

The Ingress does not change – switching still happens at the Service. Traefik picks up the new endpoints after one or two seconds; if you still get 1.1 right afterwards, simply repeat the last curl.

Clean up:

k config set-context --current --namespace=default
k delete ns tag12

Save your progress? Sign up for free

Exam Tip

  • Blue/Green = change the Service’s selector (k set selector, k patch svc, k edit svc). Canary = shared label in the Service selector, share via replicas.
  • Check whether pod labels and selector match: k get pods --show-labels, k get endpointslices -l kubernetes.io/service-name=<svc>.
  • Change labels in the template, not just in the Deployment’s metadata – and a Deployment’s selector.matchLabels is immutable after creation.

Enterprise Note

  • A canary via replica count is coarse (1 of 5 = 20 %). Finer weights come from the ingress controller: Traefik with a TraefikService of type weighted, Gateway API with backendRefs[].weight.
  • Argo Rollouts and Flagger automate canary stages and abort on their own when metrics are bad.
  • With Blue/Green, keep database schemas backward compatible – otherwise there is no way back.
  • Consider sessions (sticky sessions, caches) when switching.