Skip to content
Hands-on IT IT Drills

Day 2: Deployments

Goal: Replace the single Pod with a Deployment with three replicas, watch self-healing, roll out notes-api 0.2.0 without downtime and roll it back again. · Duration: approx. 75 min · Prerequisite: Day 1

Briefing

  • A Deployment describes the desired state: which Pod template, how many replicas, which update strategy. It doesn’t create Pods itself.
  • Instead, it creates one ReplicaSet per template version, and that keeps the number of Pods. If one is missing, it creates a new one – that is self-healing.
  • What belongs to what is decided by labels and selectors, not by names.
  • A new image means a new ReplicaSet: the Deployment scales the new one up and the old one down (rolling update). The old one stays around for a rollback.
 Deployment api          desired: 3 × notes-api:0.2.0, maxSurge 1, maxUnavailable 0
   ├── ReplicaSet api-5cbdddf87   (0.1.0)  0 Pods – kept for rollback
   └── ReplicaSet api-9cb44cd5b   (0.2.0)  3 Pods
         ├── Pod api-9cb44cd5b-vbqfv
         ├── Pod api-9cb44cd5b-6fw7j
         └── Pod api-9cb44cd5b-59nzw

Drill 1: Start of the Day, Replace the Pod

source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag02 && cd ~/k8s-lab/tag02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag01/notes/." .
kubectl delete -f k8s/api-pod.yaml
rm k8s/api-pod.yaml
ls k8s

Expected (abridged):

pod "api" deleted from notes namespace
namespace.yaml

kubectl delete -f deletes exactly the objects that are in the file. You no longer need the file api-pod.yaml from today on.

Save your progress? Sign up for free

Drill 2: The Deployment

First with the version that is already running – 0.2.0 comes in Drill 7:

cat > k8s/api.yaml <<'EOF'
# k8s/api.yaml – state at end of Day 2
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    app.kubernetes.io/part-of: notes
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: api
        app.kubernetes.io/part-of: notes
    spec:
      containers:
        - name: api
          image: notes-api:0.1.0
          env:
            - name: NOTES_API_KEY
              value: lab-api-key-123
          ports:
            - containerPort: 8080
EOF
kubectl apply --dry-run=server -f k8s/api.yaml
kubectl apply -f k8s/api.yaml
kubectl rollout status deploy/api

Expected:

deployment.apps/api created (server dry run)
deployment.apps/api created
Waiting for deployment "api" rollout to finish: 0 out of 3 new replicas have been upd…
Waiting for deployment "api" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "api" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "api" rollout to finish: 2 of 3 updated replicas are available...
deployment "api" successfully rolled out
  • template is a Pod manifest without apiVersion and kind – your Pod from yesterday, indented.
  • selector.matchLabels must match the labels in the template, otherwise the API server rejects the Deployment. Once created, the selector is immutable.
  • revisionHistoryLimit: 5 keeps five old ReplicaSets for rollbacks.

Save your progress? Sign up for free

Drill 3: Deployment, ReplicaSet, Pod

kubectl get deploy,rs,pods
kubectl get pods -o wide
POD=$(kubectl get pods -l app.kubernetes.io/name=api -o name | head -1)
RS=$(kubectl get rs -o name | head -1)
O='.metadata.ownerReferences[0]'
kubectl get "$POD" "$RS" -o custom-columns="NAME:.metadata.name,OWNER:$O.kind,:$O.name"

Expected (abridged, names, IPs and times will differ on your machine):

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/api   3/3     3            3           4s

NAME                            DESIRED   CURRENT   READY   AGE
replicaset.apps/api-5cbdddf87   3         3         3       4s

NAME                      READY   STATUS    RESTARTS   AGE
pod/api-5cbdddf87-7fjcd   1/1     Running   0          4s
pod/api-5cbdddf87-rg9q5   1/1     Running   0          4s
pod/api-5cbdddf87-thwdg   1/1     Running   0          4s
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE          …
api-5cbdddf87-7fjcd   1/1     Running   0          4s    10.244.2.4   lab-worker2   …
api-5cbdddf87-rg9q5   1/1     Running   0          4s    10.244.1.3   lab-worker    …
api-5cbdddf87-thwdg   1/1     Running   0          4s    10.244.1.4   lab-worker    …
NAME                  OWNER
api-5cbdddf87-7fjcd   ReplicaSet   api-5cbdddf87
api-5cbdddf87         Deployment   api

The names tell the chain: Deployment api, ReplicaSet api-<hash>, Pod api-<hash>-<random>. The ownerReferences say the same in machine-readable form – that is how Kubernetes cleans up everything below a Deployment when you delete it. The Pods spread across both workers; how many go where is decided by the scheduler.

Save your progress? Sign up for free

Drill 4: Self-Healing

kubectl delete "$POD"
kubectl get pods
kubectl describe "$RS" | sed -n '/^Events:/,$p'

Expected (abridged):

pod "api-5cbdddf87-7fjcd" deleted from notes namespace
NAME                  READY   STATUS              RESTARTS   AGE
api-5cbdddf87-rg9q5   1/1     Running             0          15s
api-5cbdddf87-thwdg   1/1     Running             0          15s
api-5cbdddf87-v8g2l   0/1     ContainerCreating   0          1s
Events:
  Type    Reason            Age  From                   Message
  ----    ------            ---- ----                   -------
  Normal  SuccessfulCreate  16s  replicaset-controller  Created pod: api-5cbdddf87-rg9q5
  Normal  SuccessfulCreate  15s  replicaset-controller  Created pod: api-5cbdddf87-7fjcd
  Normal  SuccessfulCreate  15s  replicaset-controller  Created pod: api-5cbdddf87-thwdg
  Normal  SuccessfulCreate  2s   replicaset-controller  Created pod: api-5cbdddf87-v8g2l

This is the core of Kubernetes: a controller constantly compares desired (3) and actual (2) and acts. Yesterday the Pod was gone after deletion; today a new one is there a second later – with a new name and a new IP.

Save your progress? Sign up for free

Drill 5: Labels and Selectors

kubectl get pods --show-labels
POD=$(kubectl get pods -l app.kubernetes.io/name=api -o name | head -1)
kubectl label "$POD" app.kubernetes.io/name=quarantaene --overwrite
sleep 2
kubectl get pods -L app.kubernetes.io/name
kubectl get "$POD" -o jsonpath='Besitzer: {.metadata.ownerReferences}'; echo

Expected (abridged):

NAME                  READY   STATUS    RESTARTS   AGE   LABELS
api-5cbdddf87-rg9q5   1/1     Running   0          16s   app.kubernetes.io/name=api,app…
api-5cbdddf87-thwdg   1/1     Running   0          16s   app.kubernetes.io/name=api,app…
api-5cbdddf87-v8g2l   1/1     Running   0          2s    app.kubernetes.io/name=api,app…
pod/api-5cbdddf87-rg9q5 labeled
NAME                  READY   STATUS    RESTARTS   AGE   NAME
api-5cbdddf87-7dfzn   1/1     Running   0          2s    api
api-5cbdddf87-rg9q5   1/1     Running   0          19s   quarantaene
api-5cbdddf87-thwdg   1/1     Running   0          19s   api
api-5cbdddf87-v8g2l   1/1     Running   0          5s    api
Besitzer:

The ReplicaSet only counts Pods that its selector matches (app.kubernetes.io/name=api,pod-template-hash=5cbdddf87). With the changed label, the Pod drops out, loses its owner (Besitzer) and keeps running on its own; the ReplicaSet immediately creates a replacement. This is how you take a suspicious Pod out of traffic for investigation. Clean up:

kubectl delete pods -l app.kubernetes.io/name=quarantaene

Expected (abridged):

pod "api-5cbdddf87-rg9q5" deleted from notes namespace

Save your progress? Sign up for free

Drill 6: Scaling and Drift

kubectl scale deploy api --replicas 5
kubectl rollout status deploy/api
kubectl get deploy api
kubectl diff -f k8s/api.yaml | grep -E '^[-+] '

Expected (abridged):

deployment.apps/api scaled
…
deployment "api" successfully rolled out
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    5/5     5            5           29s
-  generation: 2
+  generation: 3
-  replicas: 5
+  replicas: 3

kubectl scale changes the cluster, not your file. kubectl diff shows what an apply would change – here, the replicas back to 3. Exactly that:

kubectl apply -f k8s/api.yaml
sleep 2
kubectl get deploy api

Expected (abridged):

deployment.apps/api configured
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    3/3     3            3           33s

The file is the truth. Whatever you change imperatively is overwritten by the next apply.

Save your progress? Sign up for free

Drill 7: Rolling Update to 0.2.0

Build version 0.2.0 and load it into all nodes:

for app in api worker web; do
  docker build -q --build-arg VERSION=0.2.0 -t "notes-$app:0.2.0" "$NOTES/notes-$app"
  kind load docker-image "notes-$app:0.2.0" --name lab
done

Expected (abridged, IDs differ):

sha256:9c6b9f375ac7…
Image: "notes-api:0.2.0" with ID "sha256:9c6b9f375ac7…" not yet present on node
"lab-worker2", loading...
…
sha256:4e834dcf7ad7…
…

Open a second terminal and watch the Pods there:

kubectl get pods -l app.kubernetes.io/name=api -w

In the first terminal, bump the image in the file and apply it:

sed -i.bak 's/notes-api:0.1.0/notes-api:0.2.0/' k8s/api.yaml && rm k8s/api.yaml.bak
grep 'image:' k8s/api.yaml
kubectl apply -f k8s/api.yaml
kubectl rollout status deploy/api

Expected in the first terminal (abridged):

          image: notes-api:0.2.0
deployment.apps/api configured
Waiting for deployment "api" rollout to finish: 1 out of 3 new replicas have been upd…
Waiting for deployment "api" rollout to finish: 2 out of 3 new replicas have been upd…
Waiting for deployment "api" rollout to finish: 1 old replicas are pending terminati…
deployment "api" successfully rolled out

Expected in the second terminal (abridged, stop with Ctrl-C):

api-9cb44cd5b-vbqfv   0/1     Pending             0          0s
api-9cb44cd5b-vbqfv   0/1     ContainerCreating   0          0s
api-9cb44cd5b-vbqfv   1/1     Running             0          1s
api-5cbdddf87-thwdg   1/1     Terminating         0          115s
api-9cb44cd5b-6fw7j   0/1     Pending             0          0s
api-9cb44cd5b-6fw7j   1/1     Running             0          1s
api-5cbdddf87-tw9mq   1/1     Terminating         0          92s
api-9cb44cd5b-59nzw   0/1     Pending             0          0s
api-9cb44cd5b-59nzw   1/1     Running             0          1s
api-5cbdddf87-7dfzn   1/1     Terminating         0          103s

Always one new Pod first, then one old one gone. That is set by the strategy:

Setting Meaning Here
maxSurge: 1 at most 1 Pod above replicas never more than 4 Pods
maxUnavailable: 0 no Pod below replicas always 3 ready Pods

The default would be 25 % each, so with 3 replicas 1 extra Pod and – rounded down – 0 fewer. With fixed numbers it is unambiguous in the file.

kubectl get rs
kubectl logs deploy/api | grep listening

Expected (abridged):

NAME            DESIRED   CURRENT   READY   AGE
api-5cbdddf87   0         0         0       2m2s
api-9cb44cd5b   3         3         3       8s
Found 3 pods, using pod/api-9cb44cd5b-vbqfv
time=… level=INFO msg=listening addr=:8080 version=0.2.0 pid=1

The old ReplicaSet stands by with 0 Pods.

Warning: A rollout only starts when the Pod template changes. If you rebuild an image under the same tag and load it, the Deployment notices nothing – one more reason for a new version number with every build.

Save your progress? Sign up for free

Drill 8: History and Rollback

kubectl rollout history deploy/api
kubectl rollout history deploy/api --revision 1 | grep Image
kubectl rollout undo deploy/api
kubectl rollout status deploy/api
kubectl get rs

Expected (abridged):

deployment.apps/api
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

    Image:  notes-api:0.1.0
Warning: resource deployments/api was previously managed with 'kubectl apply'. Rolling
back will not update the kubectl.kubernetes.io/last-applied-configuration annotation,
which may cause unexpected behavior on future 'kubectl apply' operations. Consider using
'kubectl apply' with your previous configuration file instead.
deployment.apps/api rolled back
…
deployment "api" successfully rolled out
NAME            DESIRED   CURRENT   READY   AGE
api-5cbdddf87   3         3         3       2m15s
api-9cb44cd5b   0         0         0       21s

The rollback is a rolling update back to the old ReplicaSet – fast, because the image is already on the nodes. The warning hits the point: your file now says something different from the cluster.

kubectl rollout history deploy/api
kubectl diff -f k8s/api.yaml | grep -E '^[-+] '

Expected (abridged):

REVISION  CHANGE-CAUSE
2         <none>
3         <none>

-  generation: 5
+  generation: 6
-        image: notes-api:0.1.0
+        image: notes-api:0.2.0

Revision 1 became revision 3 – a rollback is a new revision. In an emergency, this is how you roll back in seconds and fix the file afterwards. Here 0.2.0 was fine, so back to the state of the file:

kubectl apply -f k8s/api.yaml
kubectl rollout status deploy/api
kubectl rollout history deploy/api
kubectl diff -f k8s/api.yaml; echo "diff-Exit: $?"

Expected (abridged):

deployment.apps/api configured
…
deployment "api" successfully rolled out
deployment.apps/api
REVISION  CHANGE-CAUSE
3         <none>
4         <none>

diff-Exit: 0

Save your progress? Sign up for free

Checkpoint

kubectl get deploy api -o wide

Expected: 3/3 ready, image notes-api:0.2.0.

NAME   READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES            SELECTOR
api    3/3     3            3           7s    api          notes-api:0.2.0   app.kuber…
kubectl describe deploy api | grep RollingUpdateStrategy

Expected: at most one extra Pod, none fewer.

RollingUpdateStrategy:  0 max unavailable, 1 max surge
kubectl get rs

Expected: exactly one ReplicaSet with DESIRED 3; older ones are at 0.

NAME            DESIRED   CURRENT   READY   AGE
api-5cbdddf87   0         0         0       7s
api-9cb44cd5b   3         3         3       5s
kubectl get pod api

Expected: the single Pod from Day 1 is gone.

Error from server (NotFound): pods "api" not found
kubectl diff -f k8s/api.yaml

Expected: no output. The cluster matches the file.

Challenge

Deliberately roll out a broken version notes-api:0.3.0 (which doesn’t exist). Show that the api keeps running with three Pods in the meantime, and get back to the state of the file – without touching the file.

Hint: kubectl set image only changes the cluster; with --timeout, rollout status doesn’t wait forever.

Solution

kubectl set image deploy/api api=notes-api:0.3.0
kubectl rollout status deploy/api --timeout 30s; echo "Exit: $?"
kubectl get deploy api
kubectl get pods
kubectl rollout undo deploy/api
kubectl rollout status deploy/api
kubectl diff -f k8s/api.yaml; echo "diff-Exit: $?"

Expected (abridged):

deployment.apps/api image updated
Waiting for deployment "api" rollout to finish: 1 out of 3 new replicas have been upd…
error: timed out waiting for the condition
Exit: 1
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    3/3     1            3           3m
NAME                   READY   STATUS         RESTARTS   AGE
api-6768975f45-nfpvf   0/1     ErrImagePull   0          31s
api-9cb44cd5b-8wq98    1/1     Running        0          43s
api-9cb44cd5b-g9qfn    1/1     Running        0          40s
api-9cb44cd5b-twrzx    1/1     Running        0          45s
…
deployment.apps/api rolled back
…
deployment "api" successfully rolled out
diff-Exit: 0

With maxUnavailable: 0, the Deployment may not remove an old Pod before a new one is ready. The one new Pod hangs in ErrImagePull, the three old ones keep serving. Without intervention it stays that way; after progressDeadlineSeconds (default 600 s) the Deployment reports ProgressDeadlineExceeded, but doesn’t roll back by itself. undo goes to the previous revision, i.e. 0.2.0 – which matches the file again.

Save your progress? Sign up for free

Enterprise Note

  • In production, nobody changes images with kubectl set image. The image is in Git, and a tool like Argo CD or Flux applies it and reports drift.
  • rollout undo is the emergency switch; the permanent fix is a commit that sets the file to the good version.
  • Rolling updates need readiness probes, otherwise a Pod counts as ready as soon as the process runs – even if it doesn’t answer yet (Day 5).
  • For canary and blue-green rollouts with metric analysis there are Argo Rollouts and Flagger; the Deployment itself only knows RollingUpdate and Recreate.
  • revisionHistoryLimit limits the old ReplicaSets. They cost little, but in large clusters with many Deployments it adds up.