Skip to content
Hands-on IT IT Drills

Day 15: Observing, Debugging and API Deprecations

Goal: You read status, events and logs, look into running Pods with exec, debug, port-forward and cp, and spot deprecated APIs before they hit you. · Duration: approx. 75 min · Prerequisite: Day 14

Briefing

CKAD: Application Observability and Maintenance (15 %) – “Use built-in CLI tools to monitor Kubernetes applications”, “Utilize container logs”, “Debugging in Kubernetes”, “Understand API deprecations”.

Status Usual cause First look
ErrImagePull, ImagePullBackOff Name, tag, registry k describe pod
CrashLoopBackOff, Error App exits k logs
CreateContainerConfigError Secret/ConfigMap missing k describe pod
Pending No node fits k describe pod
mkdir -p ~/ckad/tag15 && cd ~/ckad/tag15
k create ns tag15
k config set-context --current --namespace=tag15

Drill 1: Four broken Pods

k run pull --image=172.16.0.126:30500/notes-api:9.9
k run crash --image=172.16.0.126:30500/notes-api:1.0
cat > broken.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: cfg
spec:
  containers:
  - name: notes-api
    image: 172.16.0.126:30500/notes-api:1.0
    env:
    - name: NOTES_API_KEY
      valueFrom:
        secretKeyRef:
          name: notes-key
          key: NOTES_API_KEY
---
apiVersion: v1
kind: Pod
metadata:
  name: greedy
spec:
  containers:
  - name: notes-api
    image: 172.16.0.126:30500/notes-api:1.0
    env:
    - name: NOTES_API_KEY
      value: lab-key-123
    resources:
      requests:
        cpu: "8"
EOF
k apply -f broken.yaml
sleep 30
k get pods

Expected:

NAME     READY   STATUS                       RESTARTS      AGE
cfg      0/1     CreateContainerConfigError   0             30s
crash    0/1     Error                        2 (28s ago)   31s
greedy   0/1     Pending                      0             30s
pull     0/1     ErrImagePull                 0             31s

ErrImagePull turns into ImagePullBackOff after the second attempt, Error alternates with CrashLoopBackOff – the kubelet waits longer and longer between attempts.

Save your progress? Sign up for free

Drill 2: Reading the causes

k describe pod pull | grep Failed | tail -2
k logs crash
k describe pod cfg | grep Warning | tail -1
k describe pod greedy | grep FailedScheduling

Expected (abridged):

  Warning  Failed  16s (x2 over 30s)  kubelet  spec.containers{pull}: Failed to pull
image "172.16.0.126:30500/notes-api:9.9": rpc error: code = NotFound ... not found
  Warning  Failed  4s (x3 over 29s)  kubelet  spec.containers{pull}: Error:
ImagePullBackOff
time=2026-11-18T15:43:51.978Z level=ERROR msg="start abgebrochen" err="NOTES_API_KEY
oder NOTES_API_KEY_FILE fehlt"
  Warning  Failed  2s (x5 over 30s)  kubelet  spec.containers{notes-api}: Error: secret
"notes-key" not found
  Warning  FailedScheduling  6s (x9 over 31s)  default-scheduler  0/3 nodes are
available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. ...

All events in the namespace, in chronological order:

k get events --sort-by=.lastTimestamp | tail -4

Expected (abridged):

4s   Normal    BackOff   pod/pull   Back-off pulling image "172.16.0.126:30500/notes-...
4s   Warning   Failed    pod/pull   Error: ImagePullBackOff
2s   Normal    Pulled    pod/cfg    Container image "172.16.0.126:30500/notes-api:1.0"…
2s   Warning   Failed    pod/cfg    Error: secret "notes-key" not found

For cfg, only the Secret is missing. The kubelet keeps trying – creating the Secret is enough:

k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123
sleep 15
k get pod cfg
k delete pod pull crash greedy --now

Expected:

secret/notes-key created
NAME   READY   STATUS    RESTARTS   AGE
cfg    1/1     Running   0          3m30s

lab-key-123 is a pure lab value.

Save your progress? Sign up for free

Drill 3: Logs of several containers

cat > notes.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        envFrom:
        - secretRef:
            name: notes-key
      - name: poller
        image: busybox:1.37
        command: ["sh", "-c", "while true; do
          echo $(date +%T) $(wget -qO- localhost:8080/healthz); sleep 5; done"]
EOF
k apply -f notes.yaml
k rollout status deployment notes
sleep 10
k logs deploy/notes
k logs deploy/notes -c poller --tail=2
k logs -l app=notes --all-containers --prefix --since=6s

Expected (abridged):

Found 2 pods, using pod/notes-647dddd895-djlkh
Defaulted container "notes-api" out of: notes-api, poller
time=2026-11-18T15:47:45.466Z level=INFO msg="api key loaded" source=env
time=2026-11-18T15:47:45.466Z level=INFO msg="store ready" store=memory
time=2026-11-18T15:47:45.468Z level=INFO msg=listening addr=:8080 version=1.0 pid=1
Found 2 pods, using pod/notes-647dddd895-djlkh
15:47:50 ok
15:47:55 ok
[pod/notes-647dddd895-djlkh/poller] 15:47:55 ok
[pod/notes-647dddd895-rwmkp/poller] 15:47:56 ok

deploy/notes picks any one Pod, -l takes all of them. --prefix tells you where a line comes from. Follow along like tail -f, stop with Ctrl+C:

k logs -f deploy/notes -c poller --tail=1

Expected: a new line 15:48:00 ok every five seconds.

Save your progress? Sign up for free

Drill 4: Logs of the previous run

cat > once.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: once
spec:
  containers:
  - name: app
    image: busybox:1.37
    command: ["sh", "-c", "if [ -f /data/ok ]; then echo zweiter Lauf; sleep 3600;
      else touch /data/ok; echo erster Lauf; exit 1; fi"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    emptyDir: {}
EOF
k apply -f once.yaml
sleep 20
k get pod once
k logs once
k logs once --previous

Expected:

NAME   READY   STATUS    RESTARTS      AGE
once   1/1     Running   1 (17s ago)   20s
zweiter Lauf
erster Lauf

--previous shows why the last container died. For a Pod stuck in CrashLoopBackOff, however, the command in the lab (containerd 2.4.2) answered with unable to retrieve container logs for containerd://… – in that case k logs without --previous returns the last finished run.

Save your progress? Sign up for free

Drill 5: Watching resources

k top node
k top pod --containers

Expected (abridged):

NAME      CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
cp1       1515m        37%      1705Mi          14%
worker1   1613m        80%      970Mi           25%
worker2   850m         42%      846Mi           22%
POD                      NAME        CPU(cores)   MEMORY(bytes)
cfg                      notes-api   1m           1Mi
notes-647dddd895-djlkh   notes-api   6m           1Mi
notes-647dddd895-djlkh   poller      4m           0Mi
...

The values come from metrics-server and lag behind by about 15 s. k top pod --sort-by=memory sorts them.

Save your progress? Sign up for free

Drill 6: Looking into the Pod – exec and debug

k exec cfg -- ps
k debug cfg --image=busybox:1.37 --target=notes-api -c dbg -- sleep 600
sleep 5
k exec cfg -c dbg -- ps

Expected (abridged):

PID   USER     TIME  COMMAND
    1 65532     0:00 notes-api
   11 65532     0:00 ps
Targeting container "notes-api". If you don't see processes from this container it
may be because the container runtime doesn't support this feature.
PID   USER     TIME  COMMAND
    1 65532     0:00 notes-api
   30 root      0:00 sleep 600
   36 root      0:00 ps

k debug attaches an ephemeral container to the running Pod; --target shares the process namespace with notes-api. For images without a shell (distroless) this is the only way in. Ephemeral containers cannot be removed – they stay in spec.ephemeralContainers until the Pod ends. For an interactive session use k debug cfg -it --image=busybox:1.37 --target=notes-api -- sh.

You look at the node itself with a debug Pod; the node’s file system is mounted under /host:

k debug node/worker1 --image=busybox:1.37 -- \
  sh -c 'grep PRETTY /host/etc/os-release; ls /host/var/log/pods | grep tag15'
sleep 8
k logs $(k get pods -o name | grep node-debugger)
k delete $(k get pods -o name | grep node-debugger)

Expected (abridged):

Creating debugging pod node-debugger-worker1-cs7bm with container debugger on node
worker1.
PRETTY_NAME="Ubuntu 24.04.5 LTS"
tag15_cfg_911cf529-7459-47cc-9cfd-0d342bf20b77
tag15_notes-647dddd895-rwmkp_0d9088c3-20a4-4b9f-8658-b911a5de16d8
pod "node-debugger-worker1-cs7bm" deleted from tag15 namespace

Save your progress? Sign up for free

Drill 7: port-forward and cp

k port-forward deploy/notes 8080:8080 > pf.log 2>&1 &
sleep 3
curl -s localhost:8080/version
curl -s -H 'X-API-Key: lab-key-123' -d '{"text":"hallo"}' localhost:8080/notes; echo
kill %1

Expected:

1.0
{"id":"66609f3f-296c-45d8-b06f-e8fb766ee333","text":"hallo"}

port-forward tunnels through the API server – without any Service or Ingress. Copying files:

echo test > hallo.txt
k cp hallo.txt cfg:/tmp/hallo.txt
k cp cfg:/tmp/hallo.txt zurueck.txt
cat zurueck.txt

Expected: twice Defaulted container "notes-api" out of: notes-api, dbg (ephem), then test. k cp needs tar in the container and does not follow symlinks (k cp cfg:/etc/os-release … fails with skipping symlink).

Save your progress? Sign up for free

Drill 8: API versions and deprecations

k api-versions | grep -E '^(apps|autoscaling|batch|networking)'
k api-resources --api-group=autoscaling
k explain hpa --api-version=autoscaling/v1 | head -3
k explain hpa --api-version=autoscaling/v2beta2

Expected:

apps/v1
autoscaling/v1
autoscaling/v2
batch/v1
networking.k8s.io/v1
NAME                       SHORTNAMES   APIVERSION       NAMESPACED   KIND
horizontalpodautoscalers   hpa          autoscaling/v2   true    HorizontalPodAutoscaler
GROUP:      autoscaling
KIND:       HorizontalPodAutoscaler
VERSION:    v1
error: couldn't find resource for "autoscaling/v2beta2,
Resource=horizontalpodautoscalers"

autoscaling/v2beta2 was removed in 1.26. An API that still exists but is deprecated is reported by the API server as a warning every time it is used:

k expose deployment notes --port=80 --target-port=8080
k get endpoints notes
k get --raw /metrics | grep '^apiserver_requested_deprecated_apis'

Expected (abridged):

service/notes exposed
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME    ENDPOINTS                                   AGE
notes   192.168.189.113:8080,192.168.235.150:8080   0s
apiserver_requested_deprecated_apis{group="",removed_release="",resource="endpoints",
subresource="",version="v1"} 1

The metric counts which deprecated APIs are used in the cluster; removed_release names the version in which they disappear (empty = not scheduled yet).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the last 3 lines of the poller container of all notes Pods, with Pod names. Target time: 20 s
  2. Which container in the namespace uses the most memory? Target time: 20 s
  3. Show only the warnings in the namespace. Target time: 15 s

Solution

k logs -l app=notes -c poller --tail=3 --prefix
k top pod --containers --sort-by=memory | head -2
k get events --field-selector type=Warning

Expected (abridged):

[pod/notes-647dddd895-pvxfh/poller] 15:53:06 ok
...
[pod/notes-647dddd895-swdz9/poller] 15:53:16 ok
POD                      NAME        CPU(cores)   MEMORY(bytes)
notes-647dddd895-swdz9   notes-api   1m           1Mi
LAST SEEN   TYPE      REASON             OBJECT       MESSAGE
86s         Warning   Failed             pod/cfg      Error: secret "notes-key" ...
77s         Warning   BackOff            pod/crash    Back-off restarting failed ...
70s         Warning   FailedScheduling   pod/greedy   0/3 nodes are available: ...
...

Events are kept for one hour – even for Pods deleted long ago.

Save your progress? Sign up for free

Checkpoint

k get pods

Expected: cfg, once and two notes Pods (2/2), all Running.

k get pod cfg -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'

Expected: dbg

Challenge

An old manifest with networking.k8s.io/v1beta1 can no longer be applied. Make it work with the kubectl-convert plugin.

cat > old-ingress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: notes
spec:
  rules:
  - host: notes.tag15.example
    http:
      paths:
      - path: /
        backend:
          serviceName: notes
          servicePort: 80
EOF
k apply -f old-ingress.yaml

Hint: The plugin is available for each Kubernetes version under dl.k8s.io.

Solution

ARCH=$(dpkg --print-architecture)
curl -sLO https://dl.k8s.io/release/v1.37.2/bin/linux/${ARCH}/kubectl-convert
curl -sLO https://dl.k8s.io/release/v1.37.2/bin/linux/${ARCH}/kubectl-convert.sha256
echo "$(cat kubectl-convert.sha256)  kubectl-convert" | sha256sum --check
sudo install -m 755 kubectl-convert /usr/local/bin/
k convert -f old-ingress.yaml --output-version networking.k8s.io/v1 > ingress.yaml
cat ingress.yaml
k apply -f ingress.yaml

Expected (abridged):

error: resource mapping not found for name: "notes" namespace: "" from
"old-ingress.yaml": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
kubectl-convert: OK
apiVersion: networking.k8s.io/v1
kind: Ingress
...
      - backend:
          service:
            name: notes
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
ingress.networking.k8s.io/notes created

k convert works because kubectl calls programs named kubectl-* in the PATH as plugins.

Clean up:

k config set-context --current --namespace=default
k delete ns tag15

Save your progress? Sign up for free

Exam Tip

  • Order: k get pods → k describe pod (events at the bottom) → k logs (possibly -c, --previous).
  • Answer “write the logs to /opt/…” with k logs … > file – and check the file.
  • The docs page “Deprecated API Migration Guide” lists what disappears in which version.

Enterprise Note

  • Collect logs centrally (Loki, Elasticsearch); on the node they rotate away.
  • Before every cluster upgrade, run apiserver_requested_deprecated_apis and tools such as pluto or kubent against Git repos and Helm releases.
  • Allow k debug with ephemeral containers via RBAC (pods/ephemeralcontainers) only for selected teams.