Skip to content
Hands-on IT IT Drills

Day 5: Probes and Lifecycle

Goal: Kubernetes knows when api and web are ready and when they are broken, and a rollout under load doesn’t cost a single request. · Duration: approx. 75 min · Prerequisite: Day 4

Briefing

  • startupProbe: Runs first, until it succeeds once. While it runs, the other two rest. If it fails too often: restart.
  • readinessProbe: If it fails, the Pod counts as not ready in the EndpointSlice (Day 3) – no traffic, but no restart.
  • livenessProbe: If it fails, the kubelet restarts the container.
  • On termination: the Pod leaves the endpoints, at the same time preStop runs, then comes SIGTERM, and after the grace period SIGKILL.
 kubectl delete pod ──► out of endpoints ┐
                    └─► preStop sleep 5 ─┴─► SIGTERM ─► "shutdown complete"
 |◄──────────── terminationGracePeriodSeconds: 30 ─────────────►| SIGKILL

Drill 1: Rollout Without Probes

Start of the day:

source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag05 && cd ~/k8s-lab/tag05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag04/notes/." .

curl generates the load itself: [1-1000] in the URL turns it into 1000 requests in a row, --rate 25/s spreads them over 40 seconds. -w writes the HTTP code for each request (000 = no answer), sort | uniq -c counts them. & sends the whole thing to the background:

curl -s -m 2 --rate 25/s -o /dev/null -w '%{http_code}\n' \
  'localhost:30080/api/notes?n=[1-1000]' | sort | uniq -c > load.txt &
sleep 2; kubectl rollout restart deploy/api deploy/web >/dev/null
wait; cat load.txt

Expected (numbers will differ on your machine):

     14 000
    984 200
      2 500

Some of the requests fail, even though maxUnavailable: 0 applies. Without a probe, a Pod counts as ready as soon as its process runs, and an old Pod still gets traffic while it is already being terminated.

Save your progress? Sign up for free

Drill 2: Three Probes for api

The complete api.yaml is in the companion code. New are these lines in the container and terminationGracePeriodSeconds in the Pod:

# k8s/api.yaml (excerpt) – state at end of Day 5
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          # … image, env, ports as on Day 4 …
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 1
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /version
              port: 8080
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              sleep:
                seconds: 5

web gets a readinessProbe (every 5 s) and a livenessProbe (every 10 s), both on /healthz port 3000. worker stays without a probe – it has no HTTP port.

Field Default Meaning
periodSeconds 10 interval between two checks
failureThreshold 3 this many failures in a row, then the probe takes effect
timeoutSeconds 1 maximum duration of one check
initialDelaySeconds 0 wait time before the first check

So the startupProbe gives api up to 30 × 1 s to start up and checks every second – ready quickly, with no liveness restarts on a slow start.

cp "$LAB/code/tag05/notes/k8s/"{api,web}.yaml k8s/
kubectl apply --dry-run=server -f k8s/ | grep configured
kubectl apply -f k8s/ | grep configured
kubectl rollout status deploy/api >/dev/null
kubectl rollout status deploy/web >/dev/null
kubectl describe pod -l app.kubernetes.io/name=api \
  | grep -E '^\s+(Liveness|Readiness|Startup):' | sort -u

Expected (abridged):

deployment.apps/api configured (server dry run)
deployment.apps/web configured (server dry run)
deployment.apps/api configured
deployment.apps/web configured
    Liveness:       http-get http://:8080/version … period=10s … failureThreshold=3
    Readiness:      http-get http://:8080/healthz … period=5s … failureThreshold=3
    Startup:        http-get http://:8080/healthz … period=1s … failureThreshold=30

lifecycle.preStop.sleep is a built-in action of the kubelet – the image doesn’t need a sleep program. For notes-api built from scratch that is crucial: exec: [sleep, "5"] would fail there.

Save your progress? Sign up for free

Drill 3: Readiness – Out of the Endpoints

Apply a faulty version whose readiness never succeeds:

kubectl patch deploy api --type json -p '[{"op":"replace",
  "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path",
  "value":"/gibtsnicht"}]'
sleep 25
kubectl get pods -l app.kubernetes.io/name=api
kubectl rollout status deploy/api --timeout 5s; echo "Exit: $?"

Expected (abridged, names, IPs and times will differ on your machine):

deployment.apps/api patched
NAME                   READY   STATUS    RESTARTS   AGE
api-55996cd6b-4gfjv    0/1     Running   0          25s
api-65f8d67cdc-4pnkk   1/1     Running   0          3m46s
api-65f8d67cdc-sr5n6   1/1     Running   0          3m55s
api-65f8d67cdc-vp6q8   1/1     Running   0          3m50s
Waiting for deployment "api" rollout to finish: 1 out of 3 new replicas have been…
error: timed out waiting for the condition
Exit: 1

The new Pod runs, but is never 1/1. Because of maxUnavailable: 0, the Deployment doesn’t remove any old one. Who gets traffic?

kubectl get endpointslice -l kubernetes.io/service-name=api -o jsonpath=\
'{range .items[*].endpoints[*]}{.targetRef.name}  ready={.conditions.ready}{"\n"}{end}'
kubectl get events --field-selector reason=Unhealthy \
  -o custom-columns=COUNT:.count,MESSAGE:.message
curl -s -o /dev/null -w '%{http_code}\n' localhost:30080/api/notes

Expected (abridged):

api-65f8d67cdc-sr5n6  ready=true
api-65f8d67cdc-vp6q8  ready=true
api-65f8d67cdc-4pnkk  ready=true
api-55996cd6b-4gfjv  ready=false
COUNT   MESSAGE
8       Readiness probe failed: HTTP probe failed with statuscode: 404
200

The EndpointSlice lists the Pod, but as ready=false – kube-proxy sends it nothing. The app notices nothing of the broken rollout. Fix it with the file:

kubectl apply -f k8s/api.yaml && kubectl rollout status deploy/api

Expected (abridged):

deployment.apps/api configured
service/api unchanged
deployment "api" successfully rolled out

Save your progress? Sign up for free

Drill 4: Liveness – Restarts Until CrashLoopBackOff

Now the liveness is broken and the readiness healthy:

kubectl patch deploy api --type json -p '[{"op":"replace",
  "path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path",
  "value":"/gibtsnicht"}]'
kubectl rollout status deploy/api
for i in 1 2 3 4 5; do sleep 30; kubectl get pods -l app.kubernetes.io/name=api \
  --no-headers; echo; done

Expected (abridged):

deployment "api" successfully rolled out
api-6c57777c8d-f2d7z   1/1   Running   1 (1s ago)   36s
api-6c57777c8d-rbcbj   1/1   Running   0            33s
api-6c57777c8d-rsh5b   1/1   Running   1 (4s ago)   40s
…
api-6c57777c8d-f2d7z   0/1   Running   4 (2s ago)    2m7s
api-6c57777c8d-rbcbj   1/1   Running   3 (29s ago)   2m4s
api-6c57777c8d-rsh5b   1/1   Running   4 (5s ago)    2m11s

api-6c57777c8d-f2d7z   0/1   CrashLoopBackOff   4 (2s ago)    2m37s
api-6c57777c8d-rbcbj   1/1   Running            4 (29s ago)   2m34s
api-6c57777c8d-rsh5b   0/1   CrashLoopBackOff   4 (5s ago)    2m41s

The rollout was successful – readiness worked, and that is all it checks. After that, the kubelet restarts every container after three failures (30 s), and before every further start it waits longer (exponential backoff). During this wait the status is CrashLoopBackOff. What do the events and the last run say?

p=$(kubectl get pod -l app.kubernetes.io/name=api -o name | head -1)
kubectl describe $p | sed -n '/^Events/,$p' | grep -E 'Unhealthy|Killing|BackOff'
kubectl describe $p | grep -A3 'Last State'
kubectl logs $p --previous | tail -2

Expected (abridged):

  Warning  Unhealthy  38s (x15 over 2m58s)  kubelet  … Liveness probe failed: HTTP pro…
  Normal   Killing    38s (x5 over 2m38s)   kubelet  … Container api failed liveness…
  Warning  BackOff    31s (x3 over 33s)     kubelet  … Back-off restarting failed co…
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
time=… level=INFO msg="signal received, shutting down"
time=… level=INFO msg="shutdown complete"

Exit 0 and a clean shutdown complete: the application was healthy, the probe was wrong. With --previous you read the logs of the previous container – after a restart, the most important flag of kubectl logs. Fix:

kubectl apply -f k8s/api.yaml && kubectl rollout status deploy/api >/dev/null
kubectl get pods -l app.kubernetes.io/name=api

Expected (abridged):

deployment.apps/api configured
service/api unchanged
NAME                   READY   STATUS        RESTARTS      AGE
api-65f8d67cdc-jmbzt   1/1     Running       0             4s
api-65f8d67cdc-tzlst   1/1     Running       0             9s
api-65f8d67cdc-xgnvq   1/1     Running       0             7s
api-6c57777c8d-f2d7z   1/1     Terminating   5 (59s ago)   3m34s
…

Save your progress? Sign up for free

Drill 5: Why Liveness on /version

kubectl exec deploy/web -- wget -qO- http://api:8080/healthz
kubectl exec deploy/web -- wget -qO- http://api:8080/version

Expected:

ok
0.2.0

Today both paths deliver the same signal. From Day 7 on, /healthz pings the database. If the liveness were on /healthz, a Postgres restart would restart all api Pods – even though they are healthy and, after the restart, would only be waiting for the database again anyway. Readiness on /healthz, on the other hand, is exactly right: no database, no traffic. Rule of thumb: liveness only checks the process itself, never its dependencies. You prove it on Day 7.

Save your progress? Sign up for free

Drill 6: Graceful Shutdown

Delete an api Pod and follow its log at the same time:

p=$(kubectl get pod -l app.kubernetes.io/name=api \
  -o jsonpath='{.items[0].metadata.name}')
kubectl logs -f "$p" --since 1s &
sleep 1; date -u +%T; kubectl delete pod "$p" --wait=false
sleep 1
f='{range .items[*].endpoints[*]}{.targetRef.name} ready={.conditions.ready}'
f+=' terminating={.conditions.terminating}{"\n"}{end}'
kubectl get endpointslice -l kubernetes.io/service-name=api -o jsonpath="$f"
kubectl wait --for=delete pod/"$p" --timeout 60s; date -u +%T
wait

Expected (abridged):

14:47:58
pod "api-65f8d67cdc-jmbzt" deleted from notes namespace
api-65f8d67cdc-tzlst ready=true terminating=false
api-65f8d67cdc-xgnvq ready=true terminating=false
api-65f8d67cdc-jmbzt ready=false terminating=true
time=2026-10-30T14:48:03.749Z level=INFO msg="signal received, shutting down"
time=2026-10-30T14:48:03.749Z level=INFO msg="shutdown complete"
pod/api-65f8d67cdc-jmbzt condition met
14:48:05

After one second the Pod is terminating and no longer ready – out of traffic. SIGTERM only comes five seconds after the deletion: that is preStop: sleep. The pause gives kube-proxy on all nodes time to update the rules before the process stops accepting. notes-api then shuts down in milliseconds. If it needed longer than 30 s (grace period, including preStop), SIGKILL would come.

Save your progress? Sign up for free

Drill 7: Rollout Under Load

The same measurement as in Drill 1, now for api:

curl -s -m 2 --rate 25/s -o /dev/null -w '%{http_code}\n' \
  'localhost:30080/api/notes?n=[1-1500]' | sort | uniq -c > load.txt &
sleep 2; kubectl rollout restart deploy/api >/dev/null
kubectl rollout status deploy/api >/dev/null
wait; cat load.txt

Expected (numbers differ):

   1500 200

Not a single request lost. Startup and readiness probes make sure that a new Pod only gets traffic once it answers. preStop makes sure that an old one only stops once nobody is talking to it anymore. Meanwhile, 1,500 requests in one minute are served alternately by three replicas.

Save your progress? Sign up for free

Checkpoint

kubectl describe deploy api | grep -E 'Startup|Readiness|Liveness'

Expected: startup and readiness on /healthz, liveness on /version.

    Liveness:   http-get http://:8080/version delay=0s timeout=1s period=10s …
    Readiness:  http-get http://:8080/healthz delay=0s timeout=1s period=5s …
    Startup:    http-get http://:8080/healthz delay=0s timeout=1s period=1s …
kubectl get deploy api -o yaml | grep -A2 'preStop:'

Expected: before stopping, the api waits 5 seconds.

          preStop:
            sleep:
              seconds: 5
kubectl describe deploy web | grep -E 'Readiness|Liveness'

Expected: web checks /healthz on port 3000.

    Liveness:   http-get http://:3000/healthz delay=0s timeout=1s period=10s …
    Readiness:  http-get http://:3000/healthz delay=0s timeout=1s period=5s …
kubectl get pods

Expected: all 1/1, RESTARTS 0 everywhere.

NAME                      READY   STATUS    RESTARTS   AGE
api-9d94fd76d-kk26s       1/1     Running   0          17s
api-9d94fd76d-rl5mf       1/1     Running   0          15s
api-9d94fd76d-txtrv       1/1     Running   0          12s
web-d746f7494-5pc5d       1/1     Running   0          16s
web-d746f7494-n2d2t       1/1     Running   0          9s
worker-866cd96f77-hhprl   1/1     Running   0          39s
kubectl exec deploy/web -- wget -qO- http://api:8080/version

Expected: the api answers.

0.2.0

Challenge

Use the curl load from Drill 1 to measure a rollout restart of web. Are requests lost? Give web the same preStop as api temporarily and measure again. Then restore the state from the companion code.

Hint: web accepts the requests from the NodePort directly – what applied to api in Drill 6 applies here just the same.

Solution

curl -s -m 2 --rate 25/s -o /dev/null -w '%{http_code}\n' \
  'localhost:30080/api/notes?n=[1-1000]' | sort | uniq -c > load.txt &
sleep 2; kubectl rollout restart deploy/web >/dev/null
kubectl rollout status deploy/web >/dev/null; wait; cat load.txt

Expected (numbers differ):

   1000 200

In this run nothing was lost: a fast machine updates the rules faster than the old web Pods end. On a busy Mac you often see a few 000 here. In k8s/web.yaml, add below the livenessProbe (same indentation):

          lifecycle:
            preStop:
              sleep:
                seconds: 5
kubectl apply -f k8s/web.yaml && kubectl rollout status deploy/web >/dev/null
curl -s -m 2 --rate 25/s -o /dev/null -w '%{http_code}\n' \
  'localhost:30080/api/notes?n=[1-1000]' | sort | uniq -c > load.txt &
sleep 2; kubectl rollout restart deploy/web >/dev/null
kubectl rollout status deploy/web >/dev/null; wait; cat load.txt

Expected (numbers differ):

deployment.apps/web configured
service/web unchanged
   1000 200

With preStop the old Pod has five seconds of lead time; even on a busy machine at most a few isolated 000 remain. Back to the state of the companion code:

cp "$LAB/code/tag05/notes/k8s/web.yaml" k8s/
kubectl apply -f k8s/web.yaml && kubectl rollout status deploy/web

Warning: Change something like this through the file, not with kubectl patch … "op":"add". kubectl apply only removes fields that it set itself (remembered in last-applied-configuration). A lifecycle added by patch survives every later apply, and kubectl diff shows nothing.

Save your progress? Sign up for free

Enterprise Note

  • Liveness probes are the most common cause of self-inflicted outages. Never check dependencies, choose generous timeouts; when in doubt, use readiness only.
  • minReadySeconds and progressDeadlineSeconds in the Deployment make rollouts more cautious and let them fail cleanly; Argo Rollouts or Flagger add canary analysis.
  • PodDisruptionBudgets protect replicas during node maintenance (drain, Day 14) – probes alone are not enough for that.
  • After SIGTERM, applications should finish serving in-flight requests and close connections. The grace period must cover preStop plus this time.
  • Behind ingress controllers and cloud load balancers, endpoint propagation often takes longer than 5 s – choose preStop accordingly.