Day 11: Rolling Updates
Goal: You roll out new versions of notes-api without downtime, control the pace, recognize a stuck rollout and go back to a specific old revision. · Duration: approx. 60 min · Prerequisite: Day 10
Briefing
CKAD: “Understand Deployments and how to perform rolling updates” (Application Deployment, 20 %).
Deployment notes ──> ReplicaSet rev 1 (1.0) 4 -> 3 -> 2 -> 1 -> 0
└──> ReplicaSet rev 2 (1.1) 0 -> 1 -> 2 -> 3 -> 4
maxSurge: how many pods ABOVE replicas are allowed
maxUnavailable: how many pods BELOW replicas may be missing
- Every change to the pod template (
spec.template) creates a new revision, i.e. a new ReplicaSet. Changingreplicasdoes not create one.
mkdir -p ~/ckad/tag11 && cd ~/ckad/tag11
k create ns tag11
k config set-context --current --namespace=tag11Drill 1: The initial Deployment
Generate a skeleton, then add env, probe and
annotation:
k create deploy notes --image=172.16.0.126:30500/notes-api:1.0 --replicas=4 \
--port=8080 $do > notes.yamlThis is what notes.yaml looks like after editing:
cat > notes.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
labels:
app: notes
annotations:
kubernetes.io/change-cause: "notes-api 1.0"
spec:
replicas: 4
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
ports:
- containerPort: 8080
env:
- name: NOTES_API_KEY
value: lab-key-123
readinessProbe:
httpGet:
path: /healthz
port: 8080
EOF
k apply -f notes.yaml
k rollout status deploy/notes
k get deploy notes -o jsonpath='{.spec.strategy}{"\n"}'Expected (shortened):
deployment.apps/notes created
Waiting for deployment "notes" rollout to finish: 0 of 4 updated replicas are ...
deployment "notes" successfully rolled out
{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}
Without a setting, RollingUpdate with 25 % each applies.
lab-key-123 is purely a lab value – without
NOTES_API_KEY, notes-api aborts at startup.
Save your progress? Sign up for free
Drill 2: Roll out a new version
You pushed the image notes-api:1.1 on Day 2. The
registry stores data in an emptyDir – if the tag is missing after a
restart, rebuild it on cp1:
curl -s 172.16.0.126:30500/v2/notes-api/tags/list
cd ~/ckad/notes-api
podman build --build-arg VERSION=1.1 -t 172.16.0.126:30500/notes-api:1.1 . | tail -1
podman push --tls-verify=false 172.16.0.126:30500/notes-api:1.1
cd ~/ckad/tag11Expected (shortened):
{"name":"notes-api","tags":["1.0","1.1"]}
08ee432421531471091915463e3a6725254b49bece2108cb6cbe808c6538e1f3
Getting image source signatures
Writing manifest to image destination
k exec deploy/notes -- wget -qO- localhost:8080/version
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.1
k annotate deploy notes kubernetes.io/change-cause="notes-api 1.1"
k rollout status deploy/notes
k get rs
k exec deploy/notes -- wget -qO- localhost:8080/versionExpected (shortened):
1.0
deployment.apps/notes image updated
deployment.apps/notes annotated
Waiting for deployment "notes" rollout to finish: 2 out of 4 new replicas have ...
Waiting for deployment "notes" rollout to finish: 1 old replicas are pending ...
deployment "notes" successfully rolled out
NAME DESIRED CURRENT READY AGE
notes-69cd648579 0 0 0 9s
notes-ddcd7b784 4 4 4 5s
1.1
The old ReplicaSet stays around with 0 pods – it is your return
ticket for undo.
Warning:
--recordis deprecated. You write the reason for a change into the annotationkubernetes.io/change-cause;rollout historydisplays it.
Save your progress? Sign up for free
Drill 3: Control the pace
Lose no capacity, at most one extra pod:
k patch deploy notes -p \
'{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
k rollout restart deploy/notes
k get rs -wExpected (shortened, Ctrl+C stops
it):
deployment.apps/notes patched
deployment.apps/notes restarted
NAME DESIRED CURRENT READY AGE
notes-69cd648579 0 0 0 11s
notes-ddcd7b784 4 4 4 7s
notes-fd4fcc568 1 1 0 1s
notes-fd4fcc568 1 1 1 3s
notes-ddcd7b784 3 3 3 9s
notes-fd4fcc568 2 2 1 3s
notes-fd4fcc568 2 2 2 6s
notes-ddcd7b784 2 2 2 12s
...
notes-fd4fcc568 4 4 4 11s
notes-ddcd7b784 0 0 0 17s
New and old ReplicaSet together never have more than 5 pods, never
fewer than 4 ready. rollout restart only changes an
annotation in the template – handy for restarting pods (e.g. after a
changed ConfigMap).
Save your progress? Sign up for free
Drill 4: History and rollback
k rollout history deploy/notes
k rollout history deploy/notes --revision=1 | grep -E 'Image|change-cause'Expected:
deployment.apps/notes
REVISION CHANGE-CAUSE
1 notes-api 1.0
2 notes-api 1.1
3 notes-api 1.1
Annotations: kubernetes.io/change-cause: notes-api 1.0
Image: 172.16.0.126:30500/notes-api:1.0
Revision 3 is the rollout restart from Drill 3: it
inherits the Deployment’s annotation, which still says “notes-api
1.1”.
Go back to revision 1 specifically:
k rollout undo deploy/notes --to-revision=1
k rollout status deploy/notes
k rollout history deploy/notes
k exec deploy/notes -- wget -qO- localhost:8080/versionExpected (shortened):
Warning: resource deployments/notes was previously managed with 'kubectl apply'.
Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration
annotation, ...
deployment.apps/notes rolled back
deployment "notes" successfully rolled out
REVISION CHANGE-CAUSE
2 notes-api 1.1
3 notes-api 1.1
4 notes-api 1.0
1.0
Revision 1 disappears and reappears as the new, highest revision –
together with its old change cause. Without --to-revision,
undo goes back exactly one revision. The warning reminds
you that notes.yaml no longer matches the cluster.
Save your progress? Sign up for free
Drill 5: Several changes, one rollout
k rollout pause deploy/notes
k set resources deploy/notes -c notes-api --requests=cpu=50m,memory=32Mi
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.1
k annotate deploy notes kubernetes.io/change-cause="1.1 mit Requests" --overwrite
k get rsExpected:
deployment.apps/notes paused
deployment.apps/notes resource requirements updated
deployment.apps/notes image updated
deployment.apps/notes annotated
NAME DESIRED CURRENT READY AGE
notes-69cd648579 4 4 4 53s
notes-ddcd7b784 0 0 0 49s
notes-fd4fcc568 0 0 0 43s
Paused: the template has changed, but no new ReplicaSet is created.
Warning: The order matters. With
set imagefirst, the intermediate template (1.1 without requests) matches exactly the ReplicaSet of revision 2. The controller adopts it as a new revision even while paused – with the old change cause.
k rollout resume deploy/notes
k rollout status deploy/notes
k rollout history deploy/notesExpected (shortened):
deployment.apps/notes resumed
deployment "notes" successfully rolled out
REVISION CHANGE-CAUSE
2 notes-api 1.1
3 notes-api 1.1
4 notes-api 1.0
5 1.1 mit Requests
Two changes, one revision.
Save your progress? Sign up for free
Drill 6: A stuck rollout
A typo in the tag – and the deadline until the error is reported shortened to 30 seconds:
k patch deploy notes -p '{"spec":{"progressDeadlineSeconds":30}}'
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.11
k rollout status deploy/notesExpected (after about 30 s, shortened):
deployment.apps/notes patched
deployment.apps/notes image updated
Waiting for deployment "notes" rollout to finish: 1 out of 4 new replicas have been ...
error: deployment "notes" exceeded its progress deadline
k get pods
k get deploy notes \
-o jsonpath='{.status.conditions[?(@.type=="Progressing")].reason}{"\n"}'Expected (shortened):
NAME READY STATUS RESTARTS AGE
notes-5876dc8f69-9q8hh 0/1 ImagePullBackOff 0 31s
notes-869b66479b-9cnfs 1/1 Running 0 43s
notes-869b66479b-b575x 1/1 Running 0 45s
notes-869b66479b-nchzx 1/1 Running 0 41s
notes-869b66479b-qr48w 1/1 Running 0 47s
ProgressDeadlineExceeded
Thanks to maxUnavailable: 0, all four old pods keep
running – users notice nothing. Kubernetes does not
roll back by itself; you do that:
k rollout undo deploy/notes
k rollout status deploy/notesExpected (warning about
last-applied-configuration shortened):
deployment.apps/notes rolled back
deployment "notes" successfully rolled out
The pod with the wrong tag disappears; the four old ones ran without interruption.
Save your progress? Sign up for free
Drill 7: Recreate
Some applications cannot tolerate two versions at the same time (e.g. schema migration). Then stop everything first, then start again:
k patch deploy notes --type=merge \
-p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.0
k get pods -wExpected (shortened, Ctrl+C stops
it):
NAME READY STATUS RESTARTS AGE
notes-869b66479b-9cnfs 1/1 Terminating 0 49s
notes-869b66479b-b575x 1/1 Terminating 0 51s
...
notes-869b66479b-9cnfs 0/1 Completed 0 50s
notes-869b66479b-b575x 0/1 Completed 0 52s
...
notes-7cd9765ffc-zrq7n 0/1 Pending 0 0s
notes-7cd9765ffc-zrq7n 0/1 ContainerCreating 0 0s
...
notes-7cd9765ffc-dmtkd 1/1 Running 0 2s
notes-7cd9765ffc-zrq7n 1/1 Running 0 2s
First all old pods are gone, then the new ones start.
rollingUpdate: null is mandatory – without it the API
server rejects the change with
spec.strategy.rollingUpdate: Forbidden. The price is a
short downtime.
Save your progress? Sign up for free
Exercise: Speed Drills
- Switch
notesto imagenotes-api:1.1and document the reason “Speed-Drill”. Target time: 30 s - Show which image revision 2 had. Target time: 20 s
- Restart all pods of
noteswithout changing anything. Target time: 10 s
Solution
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.1
k annotate deploy notes kubernetes.io/change-cause="Speed-Drill" --overwrite
k rollout history deploy/notes --revision=2 | grep Image
k rollout restart deploy/notesExpected:
deployment.apps/notes image updated
deployment.apps/notes annotated
Image: 172.16.0.126:30500/notes-api:1.1
deployment.apps/notes restarted
With Recreate, the restart restarts all pods at once –
wait for k rollout status deploy/notes before the
checkpoint.
Save your progress? Sign up for free
Checkpoint
k get deploy notes -o jsonpath='{.spec.strategy.type}{"\n"}'Expected: Recreate
k exec deploy/notes -- wget -qO- localhost:8080/versionExpected: 1.1
Challenge
Switch notes back to RollingUpdate
(maxSurge 1, maxUnavailable 0). A new pod should only count as available
after running for 15 seconds without errors.
Hint: The field for this is directly under the
Deployment’s spec.
Solution
k explain deploy.spec.minReadySeconds | grep -A3 DESCRIPTION
k patch deploy notes --type=merge -p '{"spec":{"minReadySeconds":15,"strategy":
{"type":"RollingUpdate","rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
k rollout restart deploy/notes
k get deploy notes -wExpected (shortened, Ctrl+C stops
it):
DESCRIPTION:
Minimum number of seconds for which a newly created pod should be ready
without any of its container crashing, for it to be considered available.
Defaults to 0 (pod will be considered available as soon as it is ready)
deployment.apps/notes patched
deployment.apps/notes restarted
NAME READY UP-TO-DATE AVAILABLE AGE
notes 4/4 1 4 2m54s
notes 5/4 1 4 2m56s
notes 5/4 1 5 3m11s
notes 4/4 2 4 3m11s
notes 5/4 2 4 3m13s
notes 5/4 2 5 3m28s
...
notes 4/4 4 4 4m3s
The restart makes the rule visible: every new pod is ready after 2 s
but only counts as AVAILABLE 15 s later – only then does
the next old pod go.
Clean up:
k config set-context --current --namespace=default
k delete ns tag11The image notes-api:1.1 stays in the registry – Day 12
needs it.
Save your progress? Sign up for free
Exam Tip
- Know the rollout commands by heart:
k set image,k rollout status|history|undo|pause| resume|restart. Use--to-revisiononly with a number fromhistory. - The container name in
k set imageis the one from the template (notes-api), not the Deployment name. Look it up:k get deploy notes -o jsonpath='{..containers[*].name}'. - If
k rollout statushangs, immediately runk get podsandk describe pod– usually the image tag or a missing variable.
Enterprise Note
- Rollbacks via
undoare an emergency tool; the truth lives in Git (GitOps with Argo CD or Flux), otherwise the next sync overwrites the rollback. - Deploy images by digest (
@sha256:…) instead of tag – a tag can change. progressDeadlineSecondsplus an alert onProgressDeadlineExceededdetects stuck Deployments; automatic rollbacks are provided by Argo Rollouts or Flagger.- Without a
readinessProbe, a pod counts as ready immediately – rolling updates then only seem to protect you (more on Day 14).