Skip to content
Hands-on IT IT Drills

Day 11: Rolling Updates

Goal: You roll out new versions of notes-api without downtime, control the pace, recognize a stuck rollout and go back to a specific old revision. · Duration: approx. 60 min · Prerequisite: Day 10

Briefing

CKAD: “Understand Deployments and how to perform rolling updates” (Application Deployment, 20 %).

  Deployment notes ──> ReplicaSet rev 1 (1.0)   4 -> 3 -> 2 -> 1 -> 0
                  └──> ReplicaSet rev 2 (1.1)   0 -> 1 -> 2 -> 3 -> 4
  maxSurge:       how many pods ABOVE replicas are allowed
  maxUnavailable: how many pods BELOW replicas may be missing
  • Every change to the pod template (spec.template) creates a new revision, i.e. a new ReplicaSet. Changing replicas does not create one.
mkdir -p ~/ckad/tag11 && cd ~/ckad/tag11
k create ns tag11
k config set-context --current --namespace=tag11

Drill 1: The initial Deployment

Generate a skeleton, then add env, probe and annotation:

k create deploy notes --image=172.16.0.126:30500/notes-api:1.0 --replicas=4 \
  --port=8080 $do > notes.yaml

This is what notes.yaml looks like after editing:

cat > notes.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
  labels:
    app: notes
  annotations:
    kubernetes.io/change-cause: "notes-api 1.0"
spec:
  replicas: 4
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        ports:
        - containerPort: 8080
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
EOF
k apply -f notes.yaml
k rollout status deploy/notes
k get deploy notes -o jsonpath='{.spec.strategy}{"\n"}'

Expected (shortened):

deployment.apps/notes created
Waiting for deployment "notes" rollout to finish: 0 of 4 updated replicas are ...
deployment "notes" successfully rolled out
{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}

Without a setting, RollingUpdate with 25 % each applies. lab-key-123 is purely a lab value – without NOTES_API_KEY, notes-api aborts at startup.

Save your progress? Sign up for free

Drill 2: Roll out a new version

You pushed the image notes-api:1.1 on Day 2. The registry stores data in an emptyDir – if the tag is missing after a restart, rebuild it on cp1:

curl -s 172.16.0.126:30500/v2/notes-api/tags/list
cd ~/ckad/notes-api
podman build --build-arg VERSION=1.1 -t 172.16.0.126:30500/notes-api:1.1 . | tail -1
podman push --tls-verify=false 172.16.0.126:30500/notes-api:1.1
cd ~/ckad/tag11

Expected (shortened):

{"name":"notes-api","tags":["1.0","1.1"]}
08ee432421531471091915463e3a6725254b49bece2108cb6cbe808c6538e1f3
Getting image source signatures
Writing manifest to image destination
k exec deploy/notes -- wget -qO- localhost:8080/version
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.1
k annotate deploy notes kubernetes.io/change-cause="notes-api 1.1"
k rollout status deploy/notes
k get rs
k exec deploy/notes -- wget -qO- localhost:8080/version

Expected (shortened):

1.0
deployment.apps/notes image updated
deployment.apps/notes annotated
Waiting for deployment "notes" rollout to finish: 2 out of 4 new replicas have ...
Waiting for deployment "notes" rollout to finish: 1 old replicas are pending ...
deployment "notes" successfully rolled out
NAME               DESIRED   CURRENT   READY   AGE
notes-69cd648579   0         0         0       9s
notes-ddcd7b784    4         4         4       5s
1.1

The old ReplicaSet stays around with 0 pods – it is your return ticket for undo.

Warning: --record is deprecated. You write the reason for a change into the annotation kubernetes.io/change-cause; rollout history displays it.

Save your progress? Sign up for free

Drill 3: Control the pace

Lose no capacity, at most one extra pod:

k patch deploy notes -p \
  '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
k rollout restart deploy/notes
k get rs -w

Expected (shortened, Ctrl+C stops it):

deployment.apps/notes patched
deployment.apps/notes restarted
NAME               DESIRED   CURRENT   READY   AGE
notes-69cd648579   0         0         0       11s
notes-ddcd7b784    4         4         4       7s
notes-fd4fcc568    1         1         0       1s
notes-fd4fcc568    1         1         1       3s
notes-ddcd7b784    3         3         3       9s
notes-fd4fcc568    2         2         1       3s
notes-fd4fcc568    2         2         2       6s
notes-ddcd7b784    2         2         2       12s
...
notes-fd4fcc568    4         4         4       11s
notes-ddcd7b784    0         0         0       17s

New and old ReplicaSet together never have more than 5 pods, never fewer than 4 ready. rollout restart only changes an annotation in the template – handy for restarting pods (e.g. after a changed ConfigMap).

Save your progress? Sign up for free

Drill 4: History and rollback

k rollout history deploy/notes
k rollout history deploy/notes --revision=1 | grep -E 'Image|change-cause'

Expected:

deployment.apps/notes
REVISION  CHANGE-CAUSE
1         notes-api 1.0
2         notes-api 1.1
3         notes-api 1.1

  Annotations:  kubernetes.io/change-cause: notes-api 1.0
    Image:      172.16.0.126:30500/notes-api:1.0

Revision 3 is the rollout restart from Drill 3: it inherits the Deployment’s annotation, which still says “notes-api 1.1”.

Go back to revision 1 specifically:

k rollout undo deploy/notes --to-revision=1
k rollout status deploy/notes
k rollout history deploy/notes
k exec deploy/notes -- wget -qO- localhost:8080/version

Expected (shortened):

Warning: resource deployments/notes was previously managed with 'kubectl apply'.
Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration
annotation, ...
deployment.apps/notes rolled back
deployment "notes" successfully rolled out
REVISION  CHANGE-CAUSE
2         notes-api 1.1
3         notes-api 1.1
4         notes-api 1.0
1.0

Revision 1 disappears and reappears as the new, highest revision – together with its old change cause. Without --to-revision, undo goes back exactly one revision. The warning reminds you that notes.yaml no longer matches the cluster.

Save your progress? Sign up for free

Drill 5: Several changes, one rollout

k rollout pause deploy/notes
k set resources deploy/notes -c notes-api --requests=cpu=50m,memory=32Mi
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.1
k annotate deploy notes kubernetes.io/change-cause="1.1 mit Requests" --overwrite
k get rs

Expected:

deployment.apps/notes paused
deployment.apps/notes resource requirements updated
deployment.apps/notes image updated
deployment.apps/notes annotated
NAME               DESIRED   CURRENT   READY   AGE
notes-69cd648579   4         4         4       53s
notes-ddcd7b784    0         0         0       49s
notes-fd4fcc568    0         0         0       43s

Paused: the template has changed, but no new ReplicaSet is created.

Warning: The order matters. With set image first, the intermediate template (1.1 without requests) matches exactly the ReplicaSet of revision 2. The controller adopts it as a new revision even while paused – with the old change cause.

k rollout resume deploy/notes
k rollout status deploy/notes
k rollout history deploy/notes

Expected (shortened):

deployment.apps/notes resumed
deployment "notes" successfully rolled out
REVISION  CHANGE-CAUSE
2         notes-api 1.1
3         notes-api 1.1
4         notes-api 1.0
5         1.1 mit Requests

Two changes, one revision.

Save your progress? Sign up for free

Drill 6: A stuck rollout

A typo in the tag – and the deadline until the error is reported shortened to 30 seconds:

k patch deploy notes -p '{"spec":{"progressDeadlineSeconds":30}}'
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.11
k rollout status deploy/notes

Expected (after about 30 s, shortened):

deployment.apps/notes patched
deployment.apps/notes image updated
Waiting for deployment "notes" rollout to finish: 1 out of 4 new replicas have been ...
error: deployment "notes" exceeded its progress deadline
k get pods
k get deploy notes \
  -o jsonpath='{.status.conditions[?(@.type=="Progressing")].reason}{"\n"}'

Expected (shortened):

NAME                     READY   STATUS             RESTARTS   AGE
notes-5876dc8f69-9q8hh   0/1     ImagePullBackOff   0          31s
notes-869b66479b-9cnfs   1/1     Running            0          43s
notes-869b66479b-b575x   1/1     Running            0          45s
notes-869b66479b-nchzx   1/1     Running            0          41s
notes-869b66479b-qr48w   1/1     Running            0          47s
ProgressDeadlineExceeded

Thanks to maxUnavailable: 0, all four old pods keep running – users notice nothing. Kubernetes does not roll back by itself; you do that:

k rollout undo deploy/notes
k rollout status deploy/notes

Expected (warning about last-applied-configuration shortened):

deployment.apps/notes rolled back
deployment "notes" successfully rolled out

The pod with the wrong tag disappears; the four old ones ran without interruption.

Save your progress? Sign up for free

Drill 7: Recreate

Some applications cannot tolerate two versions at the same time (e.g. schema migration). Then stop everything first, then start again:

k patch deploy notes --type=merge \
  -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.0
k get pods -w

Expected (shortened, Ctrl+C stops it):

NAME                     READY   STATUS              RESTARTS   AGE
notes-869b66479b-9cnfs   1/1     Terminating         0          49s
notes-869b66479b-b575x   1/1     Terminating         0          51s
...
notes-869b66479b-9cnfs   0/1     Completed           0          50s
notes-869b66479b-b575x   0/1     Completed           0          52s
...
notes-7cd9765ffc-zrq7n   0/1     Pending             0          0s
notes-7cd9765ffc-zrq7n   0/1     ContainerCreating   0          0s
...
notes-7cd9765ffc-dmtkd   1/1     Running             0          2s
notes-7cd9765ffc-zrq7n   1/1     Running             0          2s

First all old pods are gone, then the new ones start. rollingUpdate: null is mandatory – without it the API server rejects the change with spec.strategy.rollingUpdate: Forbidden. The price is a short downtime.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Switch notes to image notes-api:1.1 and document the reason “Speed-Drill”. Target time: 30 s
  2. Show which image revision 2 had. Target time: 20 s
  3. Restart all pods of notes without changing anything. Target time: 10 s

Solution

k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.1
k annotate deploy notes kubernetes.io/change-cause="Speed-Drill" --overwrite
k rollout history deploy/notes --revision=2 | grep Image
k rollout restart deploy/notes

Expected:

deployment.apps/notes image updated
deployment.apps/notes annotated
    Image:      172.16.0.126:30500/notes-api:1.1
deployment.apps/notes restarted

With Recreate, the restart restarts all pods at once – wait for k rollout status deploy/notes before the checkpoint.

Save your progress? Sign up for free

Checkpoint

k get deploy notes -o jsonpath='{.spec.strategy.type}{"\n"}'

Expected: Recreate

k exec deploy/notes -- wget -qO- localhost:8080/version

Expected: 1.1

Challenge

Switch notes back to RollingUpdate (maxSurge 1, maxUnavailable 0). A new pod should only count as available after running for 15 seconds without errors.

Hint: The field for this is directly under the Deployment’s spec.

Solution

k explain deploy.spec.minReadySeconds | grep -A3 DESCRIPTION
k patch deploy notes --type=merge -p '{"spec":{"minReadySeconds":15,"strategy":
  {"type":"RollingUpdate","rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
k rollout restart deploy/notes
k get deploy notes -w

Expected (shortened, Ctrl+C stops it):

DESCRIPTION:
    Minimum number of seconds for which a newly created pod should be ready
    without any of its container crashing, for it to be considered available.
    Defaults to 0 (pod will be considered available as soon as it is ready)
deployment.apps/notes patched
deployment.apps/notes restarted
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   4/4     1            4           2m54s
notes   5/4     1            4           2m56s
notes   5/4     1            5           3m11s
notes   4/4     2            4           3m11s
notes   5/4     2            4           3m13s
notes   5/4     2            5           3m28s
...
notes   4/4     4            4           4m3s

The restart makes the rule visible: every new pod is ready after 2 s but only counts as AVAILABLE 15 s later – only then does the next old pod go.

Clean up:

k config set-context --current --namespace=default
k delete ns tag11

The image notes-api:1.1 stays in the registry – Day 12 needs it.

Save your progress? Sign up for free

Exam Tip

  • Know the rollout commands by heart: k set image, k rollout status|history|undo|pause| resume|restart. Use --to-revision only with a number from history.
  • The container name in k set image is the one from the template (notes-api), not the Deployment name. Look it up: k get deploy notes -o jsonpath='{..containers[*].name}'.
  • If k rollout status hangs, immediately run k get pods and k describe pod – usually the image tag or a missing variable.

Enterprise Note

  • Rollbacks via undo are an emergency tool; the truth lives in Git (GitOps with Argo CD or Flux), otherwise the next sync overwrites the rollback.
  • Deploy images by digest (@sha256:…) instead of tag – a tag can change.
  • progressDeadlineSeconds plus an alert on ProgressDeadlineExceeded detects stuck Deployments; automatic rollbacks are provided by Argo Rollouts or Flagger.
  • Without a readinessProbe, a pod counts as ready immediately – rolling updates then only seem to protect you (more on Day 14).