Day 5: Probes and Lifecycle
Goal: Kubernetes knows when api and web are ready and when they are broken, and a rollout under load doesn’t cost a single request. · Duration: approx. 75 min · Prerequisite: Day 4
Briefing
- startupProbe: Runs first, until it succeeds once. While it runs, the other two rest. If it fails too often: restart.
- readinessProbe: If it fails, the Pod counts as not ready in the EndpointSlice (Day 3) – no traffic, but no restart.
- livenessProbe: If it fails, the kubelet restarts the container.
- On termination: the Pod leaves the endpoints, at the same time
preStopruns, then comes SIGTERM, and after the grace period SIGKILL.
kubectl delete pod ──► out of endpoints ┐
└─► preStop sleep 5 ─┴─► SIGTERM ─► "shutdown complete"
|◄──────────── terminationGracePeriodSeconds: 30 ─────────────►| SIGKILL
Drill 1: Rollout Without Probes
Start of the day:
source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag05 && cd ~/k8s-lab/tag05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag04/notes/." .curl generates the load itself: [1-1000] in the URL
turns it into 1000 requests in a row, --rate 25/s spreads
them over 40 seconds. -w writes the HTTP code for each
request (000 = no answer), sort | uniq -c
counts them. & sends the whole thing to the
background:
curl -s -m 2 --rate 25/s -o /dev/null -w '%{http_code}\n' \
'localhost:30080/api/notes?n=[1-1000]' | sort | uniq -c > load.txt &
sleep 2; kubectl rollout restart deploy/api deploy/web >/dev/null
wait; cat load.txtExpected (numbers will differ on your machine):
14 000
984 200
2 500
Some of the requests fail, even though maxUnavailable: 0
applies. Without a probe, a Pod counts as ready as soon as its process
runs, and an old Pod still gets traffic while it is already being
terminated.
Save your progress? Sign up for free
Drill 2: Three Probes for api
The complete api.yaml is in the companion code. New are
these lines in the container and
terminationGracePeriodSeconds in the Pod:
# k8s/api.yaml (excerpt) – state at end of Day 5
spec:
terminationGracePeriodSeconds: 30
containers:
- name: api
# … image, env, ports as on Day 4 …
startupProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 1
failureThreshold: 30
readinessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
livenessProbe:
httpGet:
path: /version
port: 8080
periodSeconds: 10
failureThreshold: 3
lifecycle:
preStop:
sleep:
seconds: 5web gets a readinessProbe (every 5 s) and a
livenessProbe (every 10 s), both on /healthz
port 3000. worker stays without a probe – it has no HTTP port.
| Field | Default | Meaning |
|---|---|---|
periodSeconds |
10 | interval between two checks |
failureThreshold |
3 | this many failures in a row, then the probe takes effect |
timeoutSeconds |
1 | maximum duration of one check |
initialDelaySeconds |
0 | wait time before the first check |
So the startupProbe gives api up to 30 × 1 s to start up and checks every second – ready quickly, with no liveness restarts on a slow start.
cp "$LAB/code/tag05/notes/k8s/"{api,web}.yaml k8s/
kubectl apply --dry-run=server -f k8s/ | grep configured
kubectl apply -f k8s/ | grep configured
kubectl rollout status deploy/api >/dev/null
kubectl rollout status deploy/web >/dev/null
kubectl describe pod -l app.kubernetes.io/name=api \
| grep -E '^\s+(Liveness|Readiness|Startup):' | sort -uExpected (abridged):
deployment.apps/api configured (server dry run)
deployment.apps/web configured (server dry run)
deployment.apps/api configured
deployment.apps/web configured
Liveness: http-get http://:8080/version … period=10s … failureThreshold=3
Readiness: http-get http://:8080/healthz … period=5s … failureThreshold=3
Startup: http-get http://:8080/healthz … period=1s … failureThreshold=30
lifecycle.preStop.sleep is a built-in action of the
kubelet – the image doesn’t need a sleep program. For
notes-api built from scratch that is crucial:
exec: [sleep, "5"] would fail there.
Save your progress? Sign up for free
Drill 3: Readiness – Out of the Endpoints
Apply a faulty version whose readiness never succeeds:
kubectl patch deploy api --type json -p '[{"op":"replace",
"path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path",
"value":"/gibtsnicht"}]'
sleep 25
kubectl get pods -l app.kubernetes.io/name=api
kubectl rollout status deploy/api --timeout 5s; echo "Exit: $?"Expected (abridged, names, IPs and times will differ on your machine):
deployment.apps/api patched
NAME READY STATUS RESTARTS AGE
api-55996cd6b-4gfjv 0/1 Running 0 25s
api-65f8d67cdc-4pnkk 1/1 Running 0 3m46s
api-65f8d67cdc-sr5n6 1/1 Running 0 3m55s
api-65f8d67cdc-vp6q8 1/1 Running 0 3m50s
Waiting for deployment "api" rollout to finish: 1 out of 3 new replicas have been…
error: timed out waiting for the condition
Exit: 1
The new Pod runs, but is never 1/1. Because of
maxUnavailable: 0, the Deployment doesn’t remove any old
one. Who gets traffic?
kubectl get endpointslice -l kubernetes.io/service-name=api -o jsonpath=\
'{range .items[*].endpoints[*]}{.targetRef.name} ready={.conditions.ready}{"\n"}{end}'
kubectl get events --field-selector reason=Unhealthy \
-o custom-columns=COUNT:.count,MESSAGE:.message
curl -s -o /dev/null -w '%{http_code}\n' localhost:30080/api/notesExpected (abridged):
api-65f8d67cdc-sr5n6 ready=true
api-65f8d67cdc-vp6q8 ready=true
api-65f8d67cdc-4pnkk ready=true
api-55996cd6b-4gfjv ready=false
COUNT MESSAGE
8 Readiness probe failed: HTTP probe failed with statuscode: 404
200
The EndpointSlice lists the Pod, but as ready=false –
kube-proxy sends it nothing. The app notices nothing of the broken
rollout. Fix it with the file:
kubectl apply -f k8s/api.yaml && kubectl rollout status deploy/apiExpected (abridged):
deployment.apps/api configured
service/api unchanged
deployment "api" successfully rolled out
Save your progress? Sign up for free
Drill 4: Liveness – Restarts Until
CrashLoopBackOff
Now the liveness is broken and the readiness healthy:
kubectl patch deploy api --type json -p '[{"op":"replace",
"path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path",
"value":"/gibtsnicht"}]'
kubectl rollout status deploy/api
for i in 1 2 3 4 5; do sleep 30; kubectl get pods -l app.kubernetes.io/name=api \
--no-headers; echo; doneExpected (abridged):
deployment "api" successfully rolled out
api-6c57777c8d-f2d7z 1/1 Running 1 (1s ago) 36s
api-6c57777c8d-rbcbj 1/1 Running 0 33s
api-6c57777c8d-rsh5b 1/1 Running 1 (4s ago) 40s
…
api-6c57777c8d-f2d7z 0/1 Running 4 (2s ago) 2m7s
api-6c57777c8d-rbcbj 1/1 Running 3 (29s ago) 2m4s
api-6c57777c8d-rsh5b 1/1 Running 4 (5s ago) 2m11s
api-6c57777c8d-f2d7z 0/1 CrashLoopBackOff 4 (2s ago) 2m37s
api-6c57777c8d-rbcbj 1/1 Running 4 (29s ago) 2m34s
api-6c57777c8d-rsh5b 0/1 CrashLoopBackOff 4 (5s ago) 2m41s
The rollout was successful – readiness worked, and
that is all it checks. After that, the kubelet restarts every container
after three failures (30 s), and before every further start it waits
longer (exponential backoff). During this wait the status is
CrashLoopBackOff. What do the events and the last run
say?
p=$(kubectl get pod -l app.kubernetes.io/name=api -o name | head -1)
kubectl describe $p | sed -n '/^Events/,$p' | grep -E 'Unhealthy|Killing|BackOff'
kubectl describe $p | grep -A3 'Last State'
kubectl logs $p --previous | tail -2Expected (abridged):
Warning Unhealthy 38s (x15 over 2m58s) kubelet … Liveness probe failed: HTTP pro…
Normal Killing 38s (x5 over 2m38s) kubelet … Container api failed liveness…
Warning BackOff 31s (x3 over 33s) kubelet … Back-off restarting failed co…
Last State: Terminated
Reason: Completed
Exit Code: 0
time=… level=INFO msg="signal received, shutting down"
time=… level=INFO msg="shutdown complete"
Exit 0 and a clean shutdown complete: the application
was healthy, the probe was wrong. With --previous you read
the logs of the previous container – after a restart,
the most important flag of kubectl logs. Fix:
kubectl apply -f k8s/api.yaml && kubectl rollout status deploy/api >/dev/null
kubectl get pods -l app.kubernetes.io/name=apiExpected (abridged):
deployment.apps/api configured
service/api unchanged
NAME READY STATUS RESTARTS AGE
api-65f8d67cdc-jmbzt 1/1 Running 0 4s
api-65f8d67cdc-tzlst 1/1 Running 0 9s
api-65f8d67cdc-xgnvq 1/1 Running 0 7s
api-6c57777c8d-f2d7z 1/1 Terminating 5 (59s ago) 3m34s
…
Save your progress? Sign up for free
Drill 5: Why Liveness on
/version
kubectl exec deploy/web -- wget -qO- http://api:8080/healthz
kubectl exec deploy/web -- wget -qO- http://api:8080/versionExpected:
ok
0.2.0
Today both paths deliver the same signal. From Day 7 on,
/healthz pings the database. If the liveness were on
/healthz, a Postgres restart would restart
all api Pods – even though they are healthy and, after
the restart, would only be waiting for the database again anyway.
Readiness on /healthz, on the other hand, is exactly right:
no database, no traffic. Rule of thumb: liveness only checks the process
itself, never its dependencies. You prove it on Day 7.
Save your progress? Sign up for free
Drill 6: Graceful Shutdown
Delete an api Pod and follow its log at the same time:
p=$(kubectl get pod -l app.kubernetes.io/name=api \
-o jsonpath='{.items[0].metadata.name}')
kubectl logs -f "$p" --since 1s &
sleep 1; date -u +%T; kubectl delete pod "$p" --wait=false
sleep 1
f='{range .items[*].endpoints[*]}{.targetRef.name} ready={.conditions.ready}'
f+=' terminating={.conditions.terminating}{"\n"}{end}'
kubectl get endpointslice -l kubernetes.io/service-name=api -o jsonpath="$f"
kubectl wait --for=delete pod/"$p" --timeout 60s; date -u +%T
waitExpected (abridged):
14:47:58
pod "api-65f8d67cdc-jmbzt" deleted from notes namespace
api-65f8d67cdc-tzlst ready=true terminating=false
api-65f8d67cdc-xgnvq ready=true terminating=false
api-65f8d67cdc-jmbzt ready=false terminating=true
time=2026-10-30T14:48:03.749Z level=INFO msg="signal received, shutting down"
time=2026-10-30T14:48:03.749Z level=INFO msg="shutdown complete"
pod/api-65f8d67cdc-jmbzt condition met
14:48:05
After one second the Pod is terminating and no longer
ready – out of traffic. SIGTERM only comes five seconds
after the deletion: that is preStop: sleep. The pause gives
kube-proxy on all nodes time to update the rules before the process
stops accepting. notes-api then shuts down in milliseconds. If it needed
longer than 30 s (grace period, including preStop), SIGKILL would
come.
Save your progress? Sign up for free
Drill 7: Rollout Under Load
The same measurement as in Drill 1, now for api:
curl -s -m 2 --rate 25/s -o /dev/null -w '%{http_code}\n' \
'localhost:30080/api/notes?n=[1-1500]' | sort | uniq -c > load.txt &
sleep 2; kubectl rollout restart deploy/api >/dev/null
kubectl rollout status deploy/api >/dev/null
wait; cat load.txtExpected (numbers differ):
1500 200
Not a single request lost. Startup and readiness probes make sure that a new Pod only gets traffic once it answers. preStop makes sure that an old one only stops once nobody is talking to it anymore. Meanwhile, 1,500 requests in one minute are served alternately by three replicas.
Save your progress? Sign up for free
Checkpoint
kubectl describe deploy api | grep -E 'Startup|Readiness|Liveness'Expected: startup and readiness on /healthz, liveness on
/version.
Liveness: http-get http://:8080/version delay=0s timeout=1s period=10s …
Readiness: http-get http://:8080/healthz delay=0s timeout=1s period=5s …
Startup: http-get http://:8080/healthz delay=0s timeout=1s period=1s …
kubectl get deploy api -o yaml | grep -A2 'preStop:'Expected: before stopping, the api waits 5 seconds.
preStop:
sleep:
seconds: 5
kubectl describe deploy web | grep -E 'Readiness|Liveness'Expected: web checks /healthz on port 3000.
Liveness: http-get http://:3000/healthz delay=0s timeout=1s period=10s …
Readiness: http-get http://:3000/healthz delay=0s timeout=1s period=5s …
kubectl get podsExpected: all 1/1, RESTARTS 0
everywhere.
NAME READY STATUS RESTARTS AGE
api-9d94fd76d-kk26s 1/1 Running 0 17s
api-9d94fd76d-rl5mf 1/1 Running 0 15s
api-9d94fd76d-txtrv 1/1 Running 0 12s
web-d746f7494-5pc5d 1/1 Running 0 16s
web-d746f7494-n2d2t 1/1 Running 0 9s
worker-866cd96f77-hhprl 1/1 Running 0 39s
kubectl exec deploy/web -- wget -qO- http://api:8080/versionExpected: the api answers.
0.2.0
Challenge
Use the curl load from Drill 1 to measure a
rollout restart of web. Are requests lost?
Give web the same preStop as api temporarily and measure
again. Then restore the state from the companion code.
Hint: web accepts the requests from the NodePort directly – what applied to api in Drill 6 applies here just the same.
Solution
curl -s -m 2 --rate 25/s -o /dev/null -w '%{http_code}\n' \
'localhost:30080/api/notes?n=[1-1000]' | sort | uniq -c > load.txt &
sleep 2; kubectl rollout restart deploy/web >/dev/null
kubectl rollout status deploy/web >/dev/null; wait; cat load.txtExpected (numbers differ):
1000 200
In this run nothing was lost: a fast machine updates the rules faster
than the old web Pods end. On a busy Mac you often see a few
000 here. In k8s/web.yaml, add below the
livenessProbe (same indentation):
lifecycle:
preStop:
sleep:
seconds: 5kubectl apply -f k8s/web.yaml && kubectl rollout status deploy/web >/dev/null
curl -s -m 2 --rate 25/s -o /dev/null -w '%{http_code}\n' \
'localhost:30080/api/notes?n=[1-1000]' | sort | uniq -c > load.txt &
sleep 2; kubectl rollout restart deploy/web >/dev/null
kubectl rollout status deploy/web >/dev/null; wait; cat load.txtExpected (numbers differ):
deployment.apps/web configured
service/web unchanged
1000 200
With preStop the old Pod has five seconds of lead time; even on a
busy machine at most a few isolated 000 remain. Back to the
state of the companion code:
cp "$LAB/code/tag05/notes/k8s/web.yaml" k8s/
kubectl apply -f k8s/web.yaml && kubectl rollout status deploy/webWarning: Change something like this through the file, not with
kubectl patch … "op":"add".kubectl applyonly removes fields that it set itself (remembered inlast-applied-configuration). Alifecycleadded by patch survives every laterapply, andkubectl diffshows nothing.
Save your progress? Sign up for free
Enterprise Note
- Liveness probes are the most common cause of self-inflicted outages. Never check dependencies, choose generous timeouts; when in doubt, use readiness only.
minReadySecondsandprogressDeadlineSecondsin the Deployment make rollouts more cautious and let them fail cleanly; Argo Rollouts or Flagger add canary analysis.- PodDisruptionBudgets protect replicas during node maintenance (drain, Day 14) – probes alone are not enough for that.
- After SIGTERM, applications should finish serving in-flight requests and close connections. The grace period must cover preStop plus this time.
- Behind ingress controllers and cloud load balancers, endpoint propagation often takes longer than 5 s – choose preStop accordingly.