Day 15: Observing, Debugging and API Deprecations
Goal: You read status, events and logs, look into running Pods with
exec,debug,port-forwardandcp, and spot deprecated APIs before they hit you. · Duration: approx. 75 min · Prerequisite: Day 14
Briefing
CKAD: Application Observability and Maintenance (15 %) – “Use built-in CLI tools to monitor Kubernetes applications”, “Utilize container logs”, “Debugging in Kubernetes”, “Understand API deprecations”.
| Status | Usual cause | First look |
|---|---|---|
ErrImagePull, ImagePullBackOff |
Name, tag, registry | k describe pod |
CrashLoopBackOff, Error |
App exits | k logs |
CreateContainerConfigError |
Secret/ConfigMap missing | k describe pod |
Pending |
No node fits | k describe pod |
mkdir -p ~/ckad/tag15 && cd ~/ckad/tag15
k create ns tag15
k config set-context --current --namespace=tag15Drill 1: Four broken Pods
k run pull --image=172.16.0.126:30500/notes-api:9.9
k run crash --image=172.16.0.126:30500/notes-api:1.0
cat > broken.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: cfg
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: notes-key
key: NOTES_API_KEY
---
apiVersion: v1
kind: Pod
metadata:
name: greedy
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
resources:
requests:
cpu: "8"
EOF
k apply -f broken.yaml
sleep 30
k get podsExpected:
NAME READY STATUS RESTARTS AGE
cfg 0/1 CreateContainerConfigError 0 30s
crash 0/1 Error 2 (28s ago) 31s
greedy 0/1 Pending 0 30s
pull 0/1 ErrImagePull 0 31s
ErrImagePull turns into ImagePullBackOff
after the second attempt, Error alternates with
CrashLoopBackOff – the kubelet waits longer and longer
between attempts.
Save your progress? Sign up for free
Drill 2: Reading the causes
k describe pod pull | grep Failed | tail -2
k logs crash
k describe pod cfg | grep Warning | tail -1
k describe pod greedy | grep FailedSchedulingExpected (abridged):
Warning Failed 16s (x2 over 30s) kubelet spec.containers{pull}: Failed to pull
image "172.16.0.126:30500/notes-api:9.9": rpc error: code = NotFound ... not found
Warning Failed 4s (x3 over 29s) kubelet spec.containers{pull}: Error:
ImagePullBackOff
time=2026-11-18T15:43:51.978Z level=ERROR msg="start abgebrochen" err="NOTES_API_KEY
oder NOTES_API_KEY_FILE fehlt"
Warning Failed 2s (x5 over 30s) kubelet spec.containers{notes-api}: Error: secret
"notes-key" not found
Warning FailedScheduling 6s (x9 over 31s) default-scheduler 0/3 nodes are
available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. ...
All events in the namespace, in chronological order:
k get events --sort-by=.lastTimestamp | tail -4Expected (abridged):
4s Normal BackOff pod/pull Back-off pulling image "172.16.0.126:30500/notes-...
4s Warning Failed pod/pull Error: ImagePullBackOff
2s Normal Pulled pod/cfg Container image "172.16.0.126:30500/notes-api:1.0"…
2s Warning Failed pod/cfg Error: secret "notes-key" not found
For cfg, only the Secret is missing. The kubelet keeps
trying – creating the Secret is enough:
k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123
sleep 15
k get pod cfg
k delete pod pull crash greedy --nowExpected:
secret/notes-key created
NAME READY STATUS RESTARTS AGE
cfg 1/1 Running 0 3m30s
lab-key-123 is a pure lab value.
Save your progress? Sign up for free
Drill 3: Logs of several containers
cat > notes.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 2
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
envFrom:
- secretRef:
name: notes-key
- name: poller
image: busybox:1.37
command: ["sh", "-c", "while true; do
echo $(date +%T) $(wget -qO- localhost:8080/healthz); sleep 5; done"]
EOF
k apply -f notes.yaml
k rollout status deployment notes
sleep 10
k logs deploy/notes
k logs deploy/notes -c poller --tail=2
k logs -l app=notes --all-containers --prefix --since=6sExpected (abridged):
Found 2 pods, using pod/notes-647dddd895-djlkh
Defaulted container "notes-api" out of: notes-api, poller
time=2026-11-18T15:47:45.466Z level=INFO msg="api key loaded" source=env
time=2026-11-18T15:47:45.466Z level=INFO msg="store ready" store=memory
time=2026-11-18T15:47:45.468Z level=INFO msg=listening addr=:8080 version=1.0 pid=1
Found 2 pods, using pod/notes-647dddd895-djlkh
15:47:50 ok
15:47:55 ok
[pod/notes-647dddd895-djlkh/poller] 15:47:55 ok
[pod/notes-647dddd895-rwmkp/poller] 15:47:56 ok
deploy/notes picks any one Pod, -l takes
all of them. --prefix tells you where a line comes from.
Follow along like tail -f, stop with Ctrl+C:
k logs -f deploy/notes -c poller --tail=1Expected: a new line 15:48:00 ok every
five seconds.
Save your progress? Sign up for free
Drill 4: Logs of the previous run
cat > once.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: once
spec:
containers:
- name: app
image: busybox:1.37
command: ["sh", "-c", "if [ -f /data/ok ]; then echo zweiter Lauf; sleep 3600;
else touch /data/ok; echo erster Lauf; exit 1; fi"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
emptyDir: {}
EOF
k apply -f once.yaml
sleep 20
k get pod once
k logs once
k logs once --previousExpected:
NAME READY STATUS RESTARTS AGE
once 1/1 Running 1 (17s ago) 20s
zweiter Lauf
erster Lauf
--previous shows why the last container died. For a Pod
stuck in CrashLoopBackOff, however, the command in the lab
(containerd 2.4.2) answered with
unable to retrieve container logs for containerd://… – in
that case k logs without --previous returns
the last finished run.
Save your progress? Sign up for free
Drill 5: Watching resources
k top node
k top pod --containersExpected (abridged):
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
cp1 1515m 37% 1705Mi 14%
worker1 1613m 80% 970Mi 25%
worker2 850m 42% 846Mi 22%
POD NAME CPU(cores) MEMORY(bytes)
cfg notes-api 1m 1Mi
notes-647dddd895-djlkh notes-api 6m 1Mi
notes-647dddd895-djlkh poller 4m 0Mi
...
The values come from metrics-server and lag behind by about 15 s.
k top pod --sort-by=memory sorts them.
Save your progress? Sign up for free
Drill 6: Looking into the Pod – exec and debug
k exec cfg -- ps
k debug cfg --image=busybox:1.37 --target=notes-api -c dbg -- sleep 600
sleep 5
k exec cfg -c dbg -- psExpected (abridged):
PID USER TIME COMMAND
1 65532 0:00 notes-api
11 65532 0:00 ps
Targeting container "notes-api". If you don't see processes from this container it
may be because the container runtime doesn't support this feature.
PID USER TIME COMMAND
1 65532 0:00 notes-api
30 root 0:00 sleep 600
36 root 0:00 ps
k debug attaches an ephemeral container
to the running Pod; --target shares the process namespace
with notes-api. For images without a shell (distroless)
this is the only way in. Ephemeral containers cannot be removed – they
stay in spec.ephemeralContainers until the Pod ends. For an
interactive session use
k debug cfg -it --image=busybox:1.37 --target=notes-api -- sh.
You look at the node itself with a debug Pod; the node’s file system
is mounted under /host:
k debug node/worker1 --image=busybox:1.37 -- \
sh -c 'grep PRETTY /host/etc/os-release; ls /host/var/log/pods | grep tag15'
sleep 8
k logs $(k get pods -o name | grep node-debugger)
k delete $(k get pods -o name | grep node-debugger)Expected (abridged):
Creating debugging pod node-debugger-worker1-cs7bm with container debugger on node
worker1.
PRETTY_NAME="Ubuntu 24.04.5 LTS"
tag15_cfg_911cf529-7459-47cc-9cfd-0d342bf20b77
tag15_notes-647dddd895-rwmkp_0d9088c3-20a4-4b9f-8658-b911a5de16d8
pod "node-debugger-worker1-cs7bm" deleted from tag15 namespace
Save your progress? Sign up for free
Drill 7: port-forward and cp
k port-forward deploy/notes 8080:8080 > pf.log 2>&1 &
sleep 3
curl -s localhost:8080/version
curl -s -H 'X-API-Key: lab-key-123' -d '{"text":"hallo"}' localhost:8080/notes; echo
kill %1Expected:
1.0
{"id":"66609f3f-296c-45d8-b06f-e8fb766ee333","text":"hallo"}
port-forward tunnels through the API server – without
any Service or Ingress. Copying files:
echo test > hallo.txt
k cp hallo.txt cfg:/tmp/hallo.txt
k cp cfg:/tmp/hallo.txt zurueck.txt
cat zurueck.txtExpected: twice
Defaulted container "notes-api" out of: notes-api, dbg (ephem),
then test. k cp needs tar in the
container and does not follow symlinks
(k cp cfg:/etc/os-release … fails with
skipping symlink).
Save your progress? Sign up for free
Drill 8: API versions and deprecations
k api-versions | grep -E '^(apps|autoscaling|batch|networking)'
k api-resources --api-group=autoscaling
k explain hpa --api-version=autoscaling/v1 | head -3
k explain hpa --api-version=autoscaling/v2beta2Expected:
apps/v1
autoscaling/v1
autoscaling/v2
batch/v1
networking.k8s.io/v1
NAME SHORTNAMES APIVERSION NAMESPACED KIND
horizontalpodautoscalers hpa autoscaling/v2 true HorizontalPodAutoscaler
GROUP: autoscaling
KIND: HorizontalPodAutoscaler
VERSION: v1
error: couldn't find resource for "autoscaling/v2beta2,
Resource=horizontalpodautoscalers"
autoscaling/v2beta2 was removed in 1.26. An API that
still exists but is deprecated is reported by the API server as a
warning every time it is used:
k expose deployment notes --port=80 --target-port=8080
k get endpoints notes
k get --raw /metrics | grep '^apiserver_requested_deprecated_apis'Expected (abridged):
service/notes exposed
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
notes 192.168.189.113:8080,192.168.235.150:8080 0s
apiserver_requested_deprecated_apis{group="",removed_release="",resource="endpoints",
subresource="",version="v1"} 1
The metric counts which deprecated APIs are used in the cluster;
removed_release names the version in which they disappear
(empty = not scheduled yet).
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the last 3 lines of the
pollercontainer of all notes Pods, with Pod names. Target time: 20 s - Which container in the namespace uses the most memory? Target time: 20 s
- Show only the warnings in the namespace. Target time: 15 s
Solution
k logs -l app=notes -c poller --tail=3 --prefix
k top pod --containers --sort-by=memory | head -2
k get events --field-selector type=WarningExpected (abridged):
[pod/notes-647dddd895-pvxfh/poller] 15:53:06 ok
...
[pod/notes-647dddd895-swdz9/poller] 15:53:16 ok
POD NAME CPU(cores) MEMORY(bytes)
notes-647dddd895-swdz9 notes-api 1m 1Mi
LAST SEEN TYPE REASON OBJECT MESSAGE
86s Warning Failed pod/cfg Error: secret "notes-key" ...
77s Warning BackOff pod/crash Back-off restarting failed ...
70s Warning FailedScheduling pod/greedy 0/3 nodes are available: ...
...
Events are kept for one hour – even for Pods deleted long ago.
Save your progress? Sign up for free
Checkpoint
k get podsExpected: cfg, once and
two notes Pods (2/2), all
Running.
k get pod cfg -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'Expected: dbg
Challenge
An old manifest with networking.k8s.io/v1beta1 can no
longer be applied. Make it work with the kubectl-convert
plugin.
cat > old-ingress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: notes
spec:
rules:
- host: notes.tag15.example
http:
paths:
- path: /
backend:
serviceName: notes
servicePort: 80
EOF
k apply -f old-ingress.yamlHint: The plugin is available for each Kubernetes
version under dl.k8s.io.
Solution
ARCH=$(dpkg --print-architecture)
curl -sLO https://dl.k8s.io/release/v1.37.2/bin/linux/${ARCH}/kubectl-convert
curl -sLO https://dl.k8s.io/release/v1.37.2/bin/linux/${ARCH}/kubectl-convert.sha256
echo "$(cat kubectl-convert.sha256) kubectl-convert" | sha256sum --check
sudo install -m 755 kubectl-convert /usr/local/bin/
k convert -f old-ingress.yaml --output-version networking.k8s.io/v1 > ingress.yaml
cat ingress.yaml
k apply -f ingress.yamlExpected (abridged):
error: resource mapping not found for name: "notes" namespace: "" from
"old-ingress.yaml": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
kubectl-convert: OK
apiVersion: networking.k8s.io/v1
kind: Ingress
...
- backend:
service:
name: notes
port:
number: 80
path: /
pathType: ImplementationSpecific
ingress.networking.k8s.io/notes created
k convert works because kubectl calls programs named
kubectl-* in the PATH as plugins.
Clean up:
k config set-context --current --namespace=default
k delete ns tag15Save your progress? Sign up for free
Exam Tip
- Order:
k get pods→k describe pod(events at the bottom) →k logs(possibly-c,--previous). - Answer “write the logs to /opt/…” with
k logs … > file– and check the file. - The docs page “Deprecated API Migration Guide” lists what disappears in which version.
Enterprise Note
- Collect logs centrally (Loki, Elasticsearch); on the node they rotate away.
- Before every cluster upgrade, run
apiserver_requested_deprecated_apisand tools such asplutoorkubentagainst Git repos and Helm releases. - Allow
k debugwith ephemeral containers via RBAC (pods/ephemeralcontainers) only for selected teams.