Day 2: Deployments
Goal: Replace the single Pod with a Deployment with three replicas, watch self-healing, roll out notes-api 0.2.0 without downtime and roll it back again. · Duration: approx. 75 min · Prerequisite: Day 1
Briefing
- A Deployment describes the desired state: which Pod template, how many replicas, which update strategy. It doesn’t create Pods itself.
- Instead, it creates one ReplicaSet per template version, and that keeps the number of Pods. If one is missing, it creates a new one – that is self-healing.
- What belongs to what is decided by labels and selectors, not by names.
- A new image means a new ReplicaSet: the Deployment scales the new one up and the old one down (rolling update). The old one stays around for a rollback.
Deployment api desired: 3 × notes-api:0.2.0, maxSurge 1, maxUnavailable 0
├── ReplicaSet api-5cbdddf87 (0.1.0) 0 Pods – kept for rollback
└── ReplicaSet api-9cb44cd5b (0.2.0) 3 Pods
├── Pod api-9cb44cd5b-vbqfv
├── Pod api-9cb44cd5b-6fw7j
└── Pod api-9cb44cd5b-59nzw
Drill 1: Start of the Day, Replace the Pod
source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag02 && cd ~/k8s-lab/tag02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag01/notes/." .
kubectl delete -f k8s/api-pod.yaml
rm k8s/api-pod.yaml
ls k8sExpected (abridged):
pod "api" deleted from notes namespace
namespace.yaml
kubectl delete -f deletes exactly the objects that are
in the file. You no longer need the file api-pod.yaml from
today on.
Save your progress? Sign up for free
Drill 2: The Deployment
First with the version that is already running – 0.2.0 comes in Drill 7:
cat > k8s/api.yaml <<'EOF'
# k8s/api.yaml – state at end of Day 2
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
labels:
app.kubernetes.io/name: api
app.kubernetes.io/part-of: notes
spec:
replicas: 3
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app.kubernetes.io/name: api
template:
metadata:
labels:
app.kubernetes.io/name: api
app.kubernetes.io/part-of: notes
spec:
containers:
- name: api
image: notes-api:0.1.0
env:
- name: NOTES_API_KEY
value: lab-api-key-123
ports:
- containerPort: 8080
EOF
kubectl apply --dry-run=server -f k8s/api.yaml
kubectl apply -f k8s/api.yaml
kubectl rollout status deploy/apiExpected:
deployment.apps/api created (server dry run)
deployment.apps/api created
Waiting for deployment "api" rollout to finish: 0 out of 3 new replicas have been upd…
Waiting for deployment "api" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "api" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "api" rollout to finish: 2 of 3 updated replicas are available...
deployment "api" successfully rolled out
templateis a Pod manifest withoutapiVersionandkind– your Pod from yesterday, indented.selector.matchLabelsmust match the labels in the template, otherwise the API server rejects the Deployment. Once created, the selector is immutable.revisionHistoryLimit: 5keeps five old ReplicaSets for rollbacks.
Save your progress? Sign up for free
Drill 3: Deployment, ReplicaSet, Pod
kubectl get deploy,rs,pods
kubectl get pods -o wide
POD=$(kubectl get pods -l app.kubernetes.io/name=api -o name | head -1)
RS=$(kubectl get rs -o name | head -1)
O='.metadata.ownerReferences[0]'
kubectl get "$POD" "$RS" -o custom-columns="NAME:.metadata.name,OWNER:$O.kind,:$O.name"Expected (abridged, names, IPs and times will differ on your machine):
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/api 3/3 3 3 4s
NAME DESIRED CURRENT READY AGE
replicaset.apps/api-5cbdddf87 3 3 3 4s
NAME READY STATUS RESTARTS AGE
pod/api-5cbdddf87-7fjcd 1/1 Running 0 4s
pod/api-5cbdddf87-rg9q5 1/1 Running 0 4s
pod/api-5cbdddf87-thwdg 1/1 Running 0 4s
NAME READY STATUS RESTARTS AGE IP NODE …
api-5cbdddf87-7fjcd 1/1 Running 0 4s 10.244.2.4 lab-worker2 …
api-5cbdddf87-rg9q5 1/1 Running 0 4s 10.244.1.3 lab-worker …
api-5cbdddf87-thwdg 1/1 Running 0 4s 10.244.1.4 lab-worker …
NAME OWNER
api-5cbdddf87-7fjcd ReplicaSet api-5cbdddf87
api-5cbdddf87 Deployment api
The names tell the chain: Deployment api, ReplicaSet
api-<hash>, Pod
api-<hash>-<random>. The
ownerReferences say the same in machine-readable form –
that is how Kubernetes cleans up everything below a Deployment when you
delete it. The Pods spread across both workers; how many go where is
decided by the scheduler.
Save your progress? Sign up for free
Drill 4: Self-Healing
kubectl delete "$POD"
kubectl get pods
kubectl describe "$RS" | sed -n '/^Events:/,$p'Expected (abridged):
pod "api-5cbdddf87-7fjcd" deleted from notes namespace
NAME READY STATUS RESTARTS AGE
api-5cbdddf87-rg9q5 1/1 Running 0 15s
api-5cbdddf87-thwdg 1/1 Running 0 15s
api-5cbdddf87-v8g2l 0/1 ContainerCreating 0 1s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 16s replicaset-controller Created pod: api-5cbdddf87-rg9q5
Normal SuccessfulCreate 15s replicaset-controller Created pod: api-5cbdddf87-7fjcd
Normal SuccessfulCreate 15s replicaset-controller Created pod: api-5cbdddf87-thwdg
Normal SuccessfulCreate 2s replicaset-controller Created pod: api-5cbdddf87-v8g2l
This is the core of Kubernetes: a controller constantly compares desired (3) and actual (2) and acts. Yesterday the Pod was gone after deletion; today a new one is there a second later – with a new name and a new IP.
Save your progress? Sign up for free
Drill 5: Labels and Selectors
kubectl get pods --show-labels
POD=$(kubectl get pods -l app.kubernetes.io/name=api -o name | head -1)
kubectl label "$POD" app.kubernetes.io/name=quarantaene --overwrite
sleep 2
kubectl get pods -L app.kubernetes.io/name
kubectl get "$POD" -o jsonpath='Besitzer: {.metadata.ownerReferences}'; echoExpected (abridged):
NAME READY STATUS RESTARTS AGE LABELS
api-5cbdddf87-rg9q5 1/1 Running 0 16s app.kubernetes.io/name=api,app…
api-5cbdddf87-thwdg 1/1 Running 0 16s app.kubernetes.io/name=api,app…
api-5cbdddf87-v8g2l 1/1 Running 0 2s app.kubernetes.io/name=api,app…
pod/api-5cbdddf87-rg9q5 labeled
NAME READY STATUS RESTARTS AGE NAME
api-5cbdddf87-7dfzn 1/1 Running 0 2s api
api-5cbdddf87-rg9q5 1/1 Running 0 19s quarantaene
api-5cbdddf87-thwdg 1/1 Running 0 19s api
api-5cbdddf87-v8g2l 1/1 Running 0 5s api
Besitzer:
The ReplicaSet only counts Pods that its selector matches
(app.kubernetes.io/name=api,pod-template-hash=5cbdddf87).
With the changed label, the Pod drops out, loses its owner
(Besitzer) and keeps running on its own; the ReplicaSet
immediately creates a replacement. This is how you take a suspicious Pod
out of traffic for investigation. Clean up:
kubectl delete pods -l app.kubernetes.io/name=quarantaeneExpected (abridged):
pod "api-5cbdddf87-rg9q5" deleted from notes namespace
Save your progress? Sign up for free
Drill 6: Scaling and Drift
kubectl scale deploy api --replicas 5
kubectl rollout status deploy/api
kubectl get deploy api
kubectl diff -f k8s/api.yaml | grep -E '^[-+] 'Expected (abridged):
deployment.apps/api scaled
…
deployment "api" successfully rolled out
NAME READY UP-TO-DATE AVAILABLE AGE
api 5/5 5 5 29s
- generation: 2
+ generation: 3
- replicas: 5
+ replicas: 3
kubectl scale changes the cluster, not your file.
kubectl diff shows what an apply would change
– here, the replicas back to 3. Exactly that:
kubectl apply -f k8s/api.yaml
sleep 2
kubectl get deploy apiExpected (abridged):
deployment.apps/api configured
NAME READY UP-TO-DATE AVAILABLE AGE
api 3/3 3 3 33s
The file is the truth. Whatever you change imperatively is
overwritten by the next apply.
Save your progress? Sign up for free
Drill 7: Rolling Update to 0.2.0
Build version 0.2.0 and load it into all nodes:
for app in api worker web; do
docker build -q --build-arg VERSION=0.2.0 -t "notes-$app:0.2.0" "$NOTES/notes-$app"
kind load docker-image "notes-$app:0.2.0" --name lab
doneExpected (abridged, IDs differ):
sha256:9c6b9f375ac7…
Image: "notes-api:0.2.0" with ID "sha256:9c6b9f375ac7…" not yet present on node
"lab-worker2", loading...
…
sha256:4e834dcf7ad7…
…
Open a second terminal and watch the Pods there:
kubectl get pods -l app.kubernetes.io/name=api -wIn the first terminal, bump the image in the file and apply it:
sed -i.bak 's/notes-api:0.1.0/notes-api:0.2.0/' k8s/api.yaml && rm k8s/api.yaml.bak
grep 'image:' k8s/api.yaml
kubectl apply -f k8s/api.yaml
kubectl rollout status deploy/apiExpected in the first terminal (abridged):
image: notes-api:0.2.0
deployment.apps/api configured
Waiting for deployment "api" rollout to finish: 1 out of 3 new replicas have been upd…
Waiting for deployment "api" rollout to finish: 2 out of 3 new replicas have been upd…
Waiting for deployment "api" rollout to finish: 1 old replicas are pending terminati…
deployment "api" successfully rolled out
Expected in the second terminal (abridged, stop with
Ctrl-C):
api-9cb44cd5b-vbqfv 0/1 Pending 0 0s
api-9cb44cd5b-vbqfv 0/1 ContainerCreating 0 0s
api-9cb44cd5b-vbqfv 1/1 Running 0 1s
api-5cbdddf87-thwdg 1/1 Terminating 0 115s
api-9cb44cd5b-6fw7j 0/1 Pending 0 0s
api-9cb44cd5b-6fw7j 1/1 Running 0 1s
api-5cbdddf87-tw9mq 1/1 Terminating 0 92s
api-9cb44cd5b-59nzw 0/1 Pending 0 0s
api-9cb44cd5b-59nzw 1/1 Running 0 1s
api-5cbdddf87-7dfzn 1/1 Terminating 0 103s
Always one new Pod first, then one old one gone. That is set by the strategy:
| Setting | Meaning | Here |
|---|---|---|
maxSurge: 1 |
at most 1 Pod above replicas |
never more than 4 Pods |
maxUnavailable: 0 |
no Pod below replicas |
always 3 ready Pods |
The default would be 25 % each, so with 3 replicas 1 extra Pod and – rounded down – 0 fewer. With fixed numbers it is unambiguous in the file.
kubectl get rs
kubectl logs deploy/api | grep listeningExpected (abridged):
NAME DESIRED CURRENT READY AGE
api-5cbdddf87 0 0 0 2m2s
api-9cb44cd5b 3 3 3 8s
Found 3 pods, using pod/api-9cb44cd5b-vbqfv
time=… level=INFO msg=listening addr=:8080 version=0.2.0 pid=1
The old ReplicaSet stands by with 0 Pods.
Warning: A rollout only starts when the Pod template changes. If you rebuild an image under the same tag and load it, the Deployment notices nothing – one more reason for a new version number with every build.
Save your progress? Sign up for free
Drill 8: History and Rollback
kubectl rollout history deploy/api
kubectl rollout history deploy/api --revision 1 | grep Image
kubectl rollout undo deploy/api
kubectl rollout status deploy/api
kubectl get rsExpected (abridged):
deployment.apps/api
REVISION CHANGE-CAUSE
1 <none>
2 <none>
Image: notes-api:0.1.0
Warning: resource deployments/api was previously managed with 'kubectl apply'. Rolling
back will not update the kubectl.kubernetes.io/last-applied-configuration annotation,
which may cause unexpected behavior on future 'kubectl apply' operations. Consider using
'kubectl apply' with your previous configuration file instead.
deployment.apps/api rolled back
…
deployment "api" successfully rolled out
NAME DESIRED CURRENT READY AGE
api-5cbdddf87 3 3 3 2m15s
api-9cb44cd5b 0 0 0 21s
The rollback is a rolling update back to the old ReplicaSet – fast, because the image is already on the nodes. The warning hits the point: your file now says something different from the cluster.
kubectl rollout history deploy/api
kubectl diff -f k8s/api.yaml | grep -E '^[-+] 'Expected (abridged):
REVISION CHANGE-CAUSE
2 <none>
3 <none>
- generation: 5
+ generation: 6
- image: notes-api:0.1.0
+ image: notes-api:0.2.0
Revision 1 became revision 3 – a rollback is a new revision. In an emergency, this is how you roll back in seconds and fix the file afterwards. Here 0.2.0 was fine, so back to the state of the file:
kubectl apply -f k8s/api.yaml
kubectl rollout status deploy/api
kubectl rollout history deploy/api
kubectl diff -f k8s/api.yaml; echo "diff-Exit: $?"Expected (abridged):
deployment.apps/api configured
…
deployment "api" successfully rolled out
deployment.apps/api
REVISION CHANGE-CAUSE
3 <none>
4 <none>
diff-Exit: 0
Save your progress? Sign up for free
Checkpoint
kubectl get deploy api -o wideExpected: 3/3 ready, image
notes-api:0.2.0.
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
api 3/3 3 3 7s api notes-api:0.2.0 app.kuber…
kubectl describe deploy api | grep RollingUpdateStrategyExpected: at most one extra Pod, none fewer.
RollingUpdateStrategy: 0 max unavailable, 1 max surge
kubectl get rsExpected: exactly one ReplicaSet with DESIRED 3; older
ones are at 0.
NAME DESIRED CURRENT READY AGE
api-5cbdddf87 0 0 0 7s
api-9cb44cd5b 3 3 3 5s
kubectl get pod apiExpected: the single Pod from Day 1 is gone.
Error from server (NotFound): pods "api" not found
kubectl diff -f k8s/api.yamlExpected: no output. The cluster matches the file.
Challenge
Deliberately roll out a broken version notes-api:0.3.0
(which doesn’t exist). Show that the api keeps running with three Pods
in the meantime, and get back to the state of the file – without
touching the file.
Hint: kubectl set image only changes
the cluster; with --timeout, rollout status
doesn’t wait forever.
Solution
kubectl set image deploy/api api=notes-api:0.3.0
kubectl rollout status deploy/api --timeout 30s; echo "Exit: $?"
kubectl get deploy api
kubectl get pods
kubectl rollout undo deploy/api
kubectl rollout status deploy/api
kubectl diff -f k8s/api.yaml; echo "diff-Exit: $?"Expected (abridged):
deployment.apps/api image updated
Waiting for deployment "api" rollout to finish: 1 out of 3 new replicas have been upd…
error: timed out waiting for the condition
Exit: 1
NAME READY UP-TO-DATE AVAILABLE AGE
api 3/3 1 3 3m
NAME READY STATUS RESTARTS AGE
api-6768975f45-nfpvf 0/1 ErrImagePull 0 31s
api-9cb44cd5b-8wq98 1/1 Running 0 43s
api-9cb44cd5b-g9qfn 1/1 Running 0 40s
api-9cb44cd5b-twrzx 1/1 Running 0 45s
…
deployment.apps/api rolled back
…
deployment "api" successfully rolled out
diff-Exit: 0
With maxUnavailable: 0, the Deployment may not remove an
old Pod before a new one is ready. The one new Pod hangs in
ErrImagePull, the three old ones keep serving. Without
intervention it stays that way; after
progressDeadlineSeconds (default 600 s) the Deployment
reports ProgressDeadlineExceeded, but doesn’t roll back by
itself. undo goes to the previous revision, i.e. 0.2.0 –
which matches the file again.
Save your progress? Sign up for free
Enterprise Note
- In production, nobody changes images with
kubectl set image. The image is in Git, and a tool like Argo CD or Flux applies it and reports drift. rollout undois the emergency switch; the permanent fix is a commit that sets the file to the good version.- Rolling updates need readiness probes, otherwise a Pod counts as ready as soon as the process runs – even if it doesn’t answer yet (Day 5).
- For canary and blue-green rollouts with metric analysis there are Argo Rollouts and Flagger; the Deployment itself only knows RollingUpdate and Recreate.
revisionHistoryLimitlimits the old ReplicaSets. They cost little, but in large clusters with many Deployments it adds up.