Day 14: Probes
Goal: You give notes-api liveness, readiness and startup probes, recognize broken probes by their restarts and empty endpoints, and fix them. · Duration: approx. 60 min · Prerequisite: Day 11
Briefing
CKAD: Application Observability and Maintenance (15 %) – “Implement probes and health checks”.
| Probe | Question | If it fails |
|---|---|---|
startupProbe |
Has the app finished starting up? | Restart; the others rest until it succeeds |
livenessProbe |
Is the app still alive? | The container is restarted |
readinessProbe |
May it receive traffic? | The Pod is dropped from the Service |
Check types: httpGet, tcpSocket,
exec, grpc.
mkdir -p ~/ckad/tag14 && cd ~/ckad/tag14
k create ns tag14
k config set-context --current --namespace=tag14
k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123lab-key-123 is a pure lab value.
Drill 1: Liveness and readiness via HTTP
There is no imperative switch for probes – they always go into the YAML:
cat > notes.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 2
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
ports:
- name: http
containerPort: 8080
envFrom:
- secretRef:
name: notes-key
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 3
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz
port: http
periodSeconds: 5
EOF
k apply -f notes.yaml
k expose deployment notes --port=80 --target-port=http
k rollout status deployment notes
k describe pod -l app=notes | grep -E 'Liveness|Readiness' | head -2Expected:
deployment "notes" successfully rolled out
Liveness: http-get http://:http/healthz delay=3s timeout=1s period=10s
successThreshold=1 failureThreshold=3
Readiness: http-get http://:http/healthz delay=0s timeout=1s period=5s
successThreshold=1 failureThreshold=3
The parameters: initialDelaySeconds waits before the
first check, periodSeconds is the interval,
timeoutSeconds the patience per check,
failureThreshold the number of consecutive failures before
Kubernetes reacts. A named port (http) survives port
changes.
Save your progress? Sign up for free
Drill 2: Liveness on the wrong path
cat > broken-live.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: broken-live
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
livenessProbe:
httpGet:
path: /health
port: 8080
periodSeconds: 3
failureThreshold: 2
EOF
k apply -f broken-live.yaml
sleep 40
k get pod broken-live
k describe pod broken-live | grep -E 'Liveness probe|Killing' | tail -2Expected (abridged):
NAME READY STATUS RESTARTS AGE
broken-live 1/1 Running 4 (1s ago) 40s
Warning Unhealthy 1s (x8 over 37s) kubelet spec.containers{notes-api}: Liveness
probe failed: HTTP probe failed with statuscode: 404
Normal Killing 1s (x4 over 34s) kubelet spec.containers{notes-api}: Container
notes-api failed liveness probe, will be restarted
The app is healthy, only the probe asks the wrong question – and Kubernetes restarts it every six seconds. Probes in a Pod are immutable, so replace it:
sed -i 's|path: /health$|path: /healthz|' broken-live.yaml
k replace --force -f broken-live.yaml
sleep 15
k get pod broken-liveExpected:
pod "broken-live" deleted from tag14 namespace
pod/broken-live replaced
NAME READY STATUS RESTARTS AGE
broken-live 1/1 Running 0 15s
Save your progress? Sign up for free
Drill 3: Readiness protects the rollout
An update with the wrong readiness port:
k patch deployment notes --type=json -p='[{"op":"replace",
"path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":8081}]'
sleep 20
k get pods -l app=notes
k get endpointslices -l kubernetes.io/service-name=notes -o jsonpath=\
'{range .items[0].endpoints[*]}{.addresses[0]} ready={.conditions.ready}{"\n"}{end}'
k get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -1Expected (abridged):
NAME READY STATUS RESTARTS AGE
notes-7b8d676995-vcpjc 0/1 Running 0 20s
notes-d95bbb4ff-d8kk5 1/1 Running 0 2m40s
notes-d95bbb4ff-sw5dw 1/1 Running 0 2m40s
192.168.235.176 ready=true
192.168.189.86 ready=true
192.168.235.180 ready=false
3s Warning Unhealthy pod/notes-7b8d676995-vcpjc Readiness probe failed: Get
"http://192.168.235.180:8081/healthz": dial tcp 192.168.235.180:8081: connect:
connection refused
The new Pod runs but never becomes ready: it gets no traffic, and the
rollout stalls instead of tearing down the old Pods.
k get endpointslices also shows addresses that are not
ready in the ENDPOINTS column – only
conditions.ready reveals the state.
k rollout undo deployment notes
k rollout status deployment notesExpected (abridged):
deployment.apps/notes rolled back
deployment "notes" successfully rolled out
Save your progress? Sign up for free
Drill 4: exec probe – taking a Pod out of the Service
notes-api ships its own check command,
notes-api healthcheck (exit code 0 = healthy). In addition,
a file /tmp/down should take the Pod out of traffic:
k patch deployment notes --type=json -p='[{"op":"replace",
"path":"/spec/template/spec/containers/0/readinessProbe",
"value":{"periodSeconds":3,"exec":{"command":["sh","-c",
"test ! -f /tmp/down && notes-api healthcheck"]}}}]'
k rollout status deployment notes
POD=$(k get pods -l app=notes -o jsonpath='{.items[0].metadata.name}')
k exec $POD -- touch /tmp/down
sleep 12
k get pods -l app=notes
k get endpointslices -l kubernetes.io/service-name=notes -o jsonpath=\
'{range .items[0].endpoints[*]}{.targetRef.name} ready={.conditions.ready}{"\n"}{end}'Expected (abridged):
deployment "notes" successfully rolled out
NAME READY STATUS RESTARTS AGE
notes-5b478d48bf-6w6k2 0/1 Running 0 18s
notes-5b478d48bf-kbh6g 1/1 Running 0 16s
notes-5b478d48bf-6w6k2 ready=false
notes-5b478d48bf-kbh6g ready=true
No restart (RESTARTS 0) – readiness only takes the Pod
out of the Service. Delete the file, and it is back:
k exec $POD -- rm /tmp/down
sleep 5
k get pods -l app=notesExpected: both Pods 1/1 Running.
Save your progress? Sign up for free
Drill 5: Startup probe with tcpSocket
An app that needs 20 seconds to start. The strict liveness probe
(failureThreshold: 1) would kill it right away – the
startup probe holds it back until the port is open:
cat > slow.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: slow
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
command: ["sh", "-c", "sleep 20; exec notes-api"]
env:
- name: NOTES_API_KEY
value: lab-key-123
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 5
failureThreshold: 6
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
failureThreshold: 1
EOF
k apply -f slow.yaml
sleep 10
k get pod slow -o jsonpath='{.status.containerStatuses[0].started}{"\n"}'
sleep 20
k get pod slow
k get events --field-selector involvedObject.name=slow,reason=UnhealthyExpected (abridged):
false
NAME READY STATUS RESTARTS AGE
slow 1/1 Running 0 31s
LAST SEEN TYPE REASON OBJECT MESSAGE
11s Warning Unhealthy pod/slow Startup probe failed: dial tcp
192.168.189.101:8080: connect: connection refused
The startup budget is failureThreshold × periodSeconds =
30 s. Set it too tight:
sed -i 's/failureThreshold: 6/failureThreshold: 2/' slow.yaml
k replace --force -f slow.yaml
sleep 70
k get pod slow
k get events --field-selector involvedObject.name=slow,reason=Killing | tail -1Expected (abridged):
NAME READY STATUS RESTARTS AGE
slow 0/1 Running 1 (30s ago) 70s
20s Normal Killing pod/slow Container notes-api failed startup probe, will be
restarted
After 10 s the start counts as failed. That the restart takes another
30 s is due to sh as PID 1: it ignores SIGTERM, so the
kubelet waits out the terminationGracePeriodSeconds. This
way the app never comes up – an endless loop.
Save your progress? Sign up for free
Drill 6: Looking up gRPC probes
notes-api does not speak gRPC; for the exam it is enough to know the field:
k explain pod.spec.containers.livenessProbe.grpc | grep -E '^ [a-z]'Expected:
mode <string>
enum: Plaintext, TLS
port <integer> -required-
service <string>
A gRPC probe calls the app’s standard health service, for example
grpc: {port: 9090}. mode
(Plaintext or TLS) is new in 1.37.
Save your progress? Sign up for free
Exercise: Speed Drills
- Create the Pod
pingwith image172.16.0.126:30500/notes-api:1.0, environment variableNOTES_API_KEY=lab-key-123and a liveness probe that runsnotes-api healthcheckevery 5 s. Target time: 90 s - Show the readiness probe of the Deployment
notesas JSON. Target time: 20 s - Find all probe failures in the namespace. Target time: 15 s
Solution
k run ping --image=172.16.0.126:30500/notes-api:1.0 \
--env=NOTES_API_KEY=lab-key-123 $do > ping.yaml
vi ping.yamlAdd this to the container block (same indentation as
image:):
livenessProbe:
exec:
command: ["notes-api", "healthcheck"]
periodSeconds: 5k apply -f ping.yaml
k get deploy notes \
-o jsonpath='{.spec.template.spec.containers[0].readinessProbe}{"\n"}'
k get events --field-selector reason=UnhealthyExpected (abridged):
pod/ping created
{"exec":{"command":["sh","-c","test ! -f /tmp/down \u0026\u0026 notes-api
healthcheck"]},
"failureThreshold":3,"periodSeconds":3,"successThreshold":1,"timeoutSeconds":1}
LAST SEEN TYPE REASON OBJECT MESSAGE
2m52s Warning Unhealthy pod/broken-live Liveness probe failed:…
2m10s Warning Unhealthy pod/notes-7b8d676995-cm8w4 Readiness probe failed…
21s Warning Unhealthy pod/slow Startup probe failed: …
Save your progress? Sign up for free
Checkpoint
k get pod ping -o jsonpath='{.spec.containers[0].livenessProbe.exec.command}{"\n"}'Expected:
["notes-api","healthcheck"]
k get pod ping broken-liveExpected: both 1/1 Running,
RESTARTS 0.
k get deploy notesExpected: READY 2/2.
Challenge
The Deployment shop is running, all Pods are
Running – but the Service does not answer.
cat > shop.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop
spec:
replicas: 2
selector:
matchLabels:
app: shop
template:
metadata:
labels:
app: shop
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
envFrom:
- secretRef:
name: notes-key
readinessProbe:
httpGet:
path: /healthz
port: 8081
EOF
k apply -f shop.yaml
k expose deployment shop --port=80 --target-port=8080
k run client --image=busybox:1.37 -- sleep 3600
k wait --for=condition=Ready pod/clientHint: Check whether the Service has any ready endpoints at all.
Solution
k exec client -- wget -qO- -T3 shop/healthz
k get pods -l app=shop
k get events --field-selector reason=Unhealthy | tail -1
k patch deployment shop --type=json -p='[{"op":"replace",
"path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":8080}]'
k rollout status deployment shop
k exec client -- wget -qO- -T3 shop/healthz; echoExpected (abridged):
wget: can't connect to remote host (10.104.183.169): Connection refused
command terminated with exit code 1
NAME READY STATUS RESTARTS AGE
shop-5cb5d49f96-2bzsm 0/1 Running 0 24s
shop-5cb5d49f96-h5gz5 0/1 Running 0 24s
8s Warning Unhealthy pod/shop-5cb5d49f96-h5gz5 Readiness probe failed: Get
"http://192.168.235.175:8081/healthz": dial tcp 192.168.235.175:8081: connect:
connection refused
deployment "shop" successfully rolled out
ok
Without ready Pods the Service has no target:
Connection refused, even though everything is
Running.
Clean up:
k config set-context --current --namespace=default
k delete ns tag14Save your progress? Sign up for free
Exam Tip
- The fastest way to get probes is to copy them from the docs page “Configure Liveness, Readiness and Startup Probes” and adjust path and port.
- Restarts without errors in the logs? Check the liveness probe.
0/1 Running? The readiness probe. - Probe fields in a running Pod cannot be changed –
k replace --force -f.
Enterprise Note
- Liveness checks only the process itself, never the database or neighboring services – otherwise a database outage restarts all Pods.
- Readiness may check dependencies; that way a Pod takes itself out of traffic when there are problems.
- Catch slow starts (JVM, migrations) with a
startupProbeinstead of a hugeinitialDelaySeconds.