Skip to content
Hands-on IT IT Drills

Day 14: Probes

Goal: You give notes-api liveness, readiness and startup probes, recognize broken probes by their restarts and empty endpoints, and fix them. · Duration: approx. 60 min · Prerequisite: Day 11

Briefing

CKAD: Application Observability and Maintenance (15 %) – “Implement probes and health checks”.

Probe Question If it fails
startupProbe Has the app finished starting up? Restart; the others rest until it succeeds
livenessProbe Is the app still alive? The container is restarted
readinessProbe May it receive traffic? The Pod is dropped from the Service

Check types: httpGet, tcpSocket, exec, grpc.

mkdir -p ~/ckad/tag14 && cd ~/ckad/tag14
k create ns tag14
k config set-context --current --namespace=tag14
k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123

lab-key-123 is a pure lab value.

Drill 1: Liveness and readiness via HTTP

There is no imperative switch for probes – they always go into the YAML:

cat > notes.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        ports:
        - name: http
          containerPort: 8080
        envFrom:
        - secretRef:
            name: notes-key
        livenessProbe:
          httpGet:
            path: /healthz
            port: http
          initialDelaySeconds: 3
          periodSeconds: 10
          timeoutSeconds: 1
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /healthz
            port: http
          periodSeconds: 5
EOF
k apply -f notes.yaml
k expose deployment notes --port=80 --target-port=http
k rollout status deployment notes
k describe pod -l app=notes | grep -E 'Liveness|Readiness' | head -2

Expected:

deployment "notes" successfully rolled out
    Liveness:   http-get http://:http/healthz delay=3s timeout=1s period=10s
successThreshold=1 failureThreshold=3
    Readiness:  http-get http://:http/healthz delay=0s timeout=1s period=5s
successThreshold=1 failureThreshold=3

The parameters: initialDelaySeconds waits before the first check, periodSeconds is the interval, timeoutSeconds the patience per check, failureThreshold the number of consecutive failures before Kubernetes reacts. A named port (http) survives port changes.

Save your progress? Sign up for free

Drill 2: Liveness on the wrong path

cat > broken-live.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: broken-live
spec:
  containers:
  - name: notes-api
    image: 172.16.0.126:30500/notes-api:1.0
    env:
    - name: NOTES_API_KEY
      value: lab-key-123
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      periodSeconds: 3
      failureThreshold: 2
EOF
k apply -f broken-live.yaml
sleep 40
k get pod broken-live
k describe pod broken-live | grep -E 'Liveness probe|Killing' | tail -2

Expected (abridged):

NAME          READY   STATUS    RESTARTS     AGE
broken-live   1/1     Running   4 (1s ago)   40s
  Warning  Unhealthy  1s (x8 over 37s)  kubelet  spec.containers{notes-api}: Liveness
probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    1s (x4 over 34s)  kubelet  spec.containers{notes-api}: Container
notes-api failed liveness probe, will be restarted

The app is healthy, only the probe asks the wrong question – and Kubernetes restarts it every six seconds. Probes in a Pod are immutable, so replace it:

sed -i 's|path: /health$|path: /healthz|' broken-live.yaml
k replace --force -f broken-live.yaml
sleep 15
k get pod broken-live

Expected:

pod "broken-live" deleted from tag14 namespace
pod/broken-live replaced
NAME          READY   STATUS    RESTARTS   AGE
broken-live   1/1     Running   0          15s

Save your progress? Sign up for free

Drill 3: Readiness protects the rollout

An update with the wrong readiness port:

k patch deployment notes --type=json -p='[{"op":"replace",
  "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":8081}]'
sleep 20
k get pods -l app=notes
k get endpointslices -l kubernetes.io/service-name=notes -o jsonpath=\
'{range .items[0].endpoints[*]}{.addresses[0]} ready={.conditions.ready}{"\n"}{end}'
k get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -1

Expected (abridged):

NAME                     READY   STATUS    RESTARTS   AGE
notes-7b8d676995-vcpjc   0/1     Running   0          20s
notes-d95bbb4ff-d8kk5    1/1     Running   0          2m40s
notes-d95bbb4ff-sw5dw    1/1     Running   0          2m40s
192.168.235.176 ready=true
192.168.189.86 ready=true
192.168.235.180 ready=false
3s   Warning   Unhealthy   pod/notes-7b8d676995-vcpjc   Readiness probe failed: Get
"http://192.168.235.180:8081/healthz": dial tcp 192.168.235.180:8081: connect:
connection refused

The new Pod runs but never becomes ready: it gets no traffic, and the rollout stalls instead of tearing down the old Pods. k get endpointslices also shows addresses that are not ready in the ENDPOINTS column – only conditions.ready reveals the state.

k rollout undo deployment notes
k rollout status deployment notes

Expected (abridged):

deployment.apps/notes rolled back
deployment "notes" successfully rolled out

Save your progress? Sign up for free

Drill 4: exec probe – taking a Pod out of the Service

notes-api ships its own check command, notes-api healthcheck (exit code 0 = healthy). In addition, a file /tmp/down should take the Pod out of traffic:

k patch deployment notes --type=json -p='[{"op":"replace",
  "path":"/spec/template/spec/containers/0/readinessProbe",
  "value":{"periodSeconds":3,"exec":{"command":["sh","-c",
  "test ! -f /tmp/down && notes-api healthcheck"]}}}]'
k rollout status deployment notes
POD=$(k get pods -l app=notes -o jsonpath='{.items[0].metadata.name}')
k exec $POD -- touch /tmp/down
sleep 12
k get pods -l app=notes
k get endpointslices -l kubernetes.io/service-name=notes -o jsonpath=\
'{range .items[0].endpoints[*]}{.targetRef.name} ready={.conditions.ready}{"\n"}{end}'

Expected (abridged):

deployment "notes" successfully rolled out
NAME                     READY   STATUS    RESTARTS   AGE
notes-5b478d48bf-6w6k2   0/1     Running   0          18s
notes-5b478d48bf-kbh6g   1/1     Running   0          16s
notes-5b478d48bf-6w6k2 ready=false
notes-5b478d48bf-kbh6g ready=true

No restart (RESTARTS 0) – readiness only takes the Pod out of the Service. Delete the file, and it is back:

k exec $POD -- rm /tmp/down
sleep 5
k get pods -l app=notes

Expected: both Pods 1/1 Running.

Save your progress? Sign up for free

Drill 5: Startup probe with tcpSocket

An app that needs 20 seconds to start. The strict liveness probe (failureThreshold: 1) would kill it right away – the startup probe holds it back until the port is open:

cat > slow.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: slow
spec:
  containers:
  - name: notes-api
    image: 172.16.0.126:30500/notes-api:1.0
    command: ["sh", "-c", "sleep 20; exec notes-api"]
    env:
    - name: NOTES_API_KEY
      value: lab-key-123
    startupProbe:
      tcpSocket:
        port: 8080
      periodSeconds: 5
      failureThreshold: 6
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 5
      failureThreshold: 1
EOF
k apply -f slow.yaml
sleep 10
k get pod slow -o jsonpath='{.status.containerStatuses[0].started}{"\n"}'
sleep 20
k get pod slow
k get events --field-selector involvedObject.name=slow,reason=Unhealthy

Expected (abridged):

false
NAME   READY   STATUS    RESTARTS   AGE
slow   1/1     Running   0          31s
LAST SEEN   TYPE      REASON      OBJECT     MESSAGE
11s         Warning   Unhealthy   pod/slow   Startup probe failed: dial tcp
192.168.189.101:8080: connect: connection refused

The startup budget is failureThreshold × periodSeconds = 30 s. Set it too tight:

sed -i 's/failureThreshold: 6/failureThreshold: 2/' slow.yaml
k replace --force -f slow.yaml
sleep 70
k get pod slow
k get events --field-selector involvedObject.name=slow,reason=Killing | tail -1

Expected (abridged):

NAME   READY   STATUS    RESTARTS      AGE
slow   0/1     Running   1 (30s ago)   70s
20s    Normal   Killing   pod/slow   Container notes-api failed startup probe, will be
restarted

After 10 s the start counts as failed. That the restart takes another 30 s is due to sh as PID 1: it ignores SIGTERM, so the kubelet waits out the terminationGracePeriodSeconds. This way the app never comes up – an endless loop.

Save your progress? Sign up for free

Drill 6: Looking up gRPC probes

notes-api does not speak gRPC; for the exam it is enough to know the field:

k explain pod.spec.containers.livenessProbe.grpc | grep -E '^  [a-z]'

Expected:

  mode  <string>
  enum: Plaintext, TLS
  port  <integer> -required-
  service   <string>

A gRPC probe calls the app’s standard health service, for example grpc: {port: 9090}. mode (Plaintext or TLS) is new in 1.37.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Create the Pod ping with image 172.16.0.126:30500/notes-api:1.0, environment variable NOTES_API_KEY=lab-key-123 and a liveness probe that runs notes-api healthcheck every 5 s. Target time: 90 s
  2. Show the readiness probe of the Deployment notes as JSON. Target time: 20 s
  3. Find all probe failures in the namespace. Target time: 15 s

Solution

k run ping --image=172.16.0.126:30500/notes-api:1.0 \
  --env=NOTES_API_KEY=lab-key-123 $do > ping.yaml
vi ping.yaml

Add this to the container block (same indentation as image:):

    livenessProbe:
      exec:
        command: ["notes-api", "healthcheck"]
      periodSeconds: 5
k apply -f ping.yaml
k get deploy notes \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe}{"\n"}'
k get events --field-selector reason=Unhealthy

Expected (abridged):

pod/ping created
{"exec":{"command":["sh","-c","test ! -f /tmp/down \u0026\u0026 notes-api
healthcheck"]},
"failureThreshold":3,"periodSeconds":3,"successThreshold":1,"timeoutSeconds":1}
LAST SEEN   TYPE      REASON      OBJECT                       MESSAGE
2m52s       Warning   Unhealthy   pod/broken-live              Liveness probe failed:…
2m10s       Warning   Unhealthy   pod/notes-7b8d676995-cm8w4   Readiness probe failed…
21s         Warning   Unhealthy   pod/slow                     Startup probe failed: …

Save your progress? Sign up for free

Checkpoint

k get pod ping -o jsonpath='{.spec.containers[0].livenessProbe.exec.command}{"\n"}'

Expected: ["notes-api","healthcheck"]

k get pod ping broken-live

Expected: both 1/1 Running, RESTARTS 0.

k get deploy notes

Expected: READY 2/2.

Challenge

The Deployment shop is running, all Pods are Running – but the Service does not answer.

cat > shop.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: shop
  template:
    metadata:
      labels:
        app: shop
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        envFrom:
        - secretRef:
            name: notes-key
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8081
EOF
k apply -f shop.yaml
k expose deployment shop --port=80 --target-port=8080
k run client --image=busybox:1.37 -- sleep 3600
k wait --for=condition=Ready pod/client

Hint: Check whether the Service has any ready endpoints at all.

Solution

k exec client -- wget -qO- -T3 shop/healthz
k get pods -l app=shop
k get events --field-selector reason=Unhealthy | tail -1
k patch deployment shop --type=json -p='[{"op":"replace",
  "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":8080}]'
k rollout status deployment shop
k exec client -- wget -qO- -T3 shop/healthz; echo

Expected (abridged):

wget: can't connect to remote host (10.104.183.169): Connection refused
command terminated with exit code 1
NAME                    READY   STATUS    RESTARTS   AGE
shop-5cb5d49f96-2bzsm   0/1     Running   0          24s
shop-5cb5d49f96-h5gz5   0/1     Running   0          24s
8s   Warning   Unhealthy   pod/shop-5cb5d49f96-h5gz5   Readiness probe failed: Get
"http://192.168.235.175:8081/healthz": dial tcp 192.168.235.175:8081: connect:
connection refused
deployment "shop" successfully rolled out
ok

Without ready Pods the Service has no target: Connection refused, even though everything is Running.

Clean up:

k config set-context --current --namespace=default
k delete ns tag14

Save your progress? Sign up for free

Exam Tip

  • The fastest way to get probes is to copy them from the docs page “Configure Liveness, Readiness and Startup Probes” and adjust path and port.
  • Restarts without errors in the logs? Check the liveness probe. 0/1 Running? The readiness probe.
  • Probe fields in a running Pod cannot be changed – k replace --force -f.

Enterprise Note

  • Liveness checks only the process itself, never the database or neighboring services – otherwise a database outage restarts all Pods.
  • Readiness may check dependencies; that way a Pod takes itself out of traffic when there are problems.
  • Catch slow starts (JVM, migrations) with a startupProbe instead of a huge initialDelaySeconds.