Day 15: Troubleshooting II – Broken Apps and Networks
Goal: From status, events, and logs you can tell in seconds why an app is not running or not reachable – and you fix it. · Duration: approx. 75 min · Prerequisite: Day 9
Briefing
CKA: Troubleshooting (30 %) – “Manage and evaluate container output streams”, “Troubleshoot services and networking”.
| Status / symptom | Usual cause | Tool |
|---|---|---|
CrashLoopBackOff, Error |
Command, config, crash | k logs |
ImagePullBackOff |
Name, tag, registry access | k describe pod |
OOMKilled, exit 137 |
Memory limit too small | k describe pod |
Service: Connection refused |
Selector, targetPort, readiness | k get endpointslices |
bad address |
DNS | Check CoreDNS |
download timed out |
NetworkPolicy | k get netpol |
k create ns tag15
k config set-context --current --namespace=tag15Drill 1: Three Broken Pods
k run crash --image=busybox:1.37 -- \
sh -c 'echo starte; echo Konfiguration fehlt >&2; exit 1'
k run pull --image=nginx:1.299
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
name: hungry
spec:
containers:
- name: app
image: polinux/stress:1.0.4
command: ["stress", "--vm", "1", "--vm-bytes", "200M", "--vm-hang", "1"]
resources:
limits:
memory: 100Mi
EOF
k get podsExpected (after about 45 s):
NAME READY STATUS RESTARTS AGE
crash 0/1 Error 3 (30s ago) 46s
hungry 0/1 OOMKilled 3 (26s ago) 45s
pull 0/1 ImagePullBackOff 0 46s
The status alternates between Error and
CrashLoopBackOff – Kubernetes waits longer and longer
between restarts (up to five minutes).
Save your progress? Sign up for free
Drill 2: Logs of a Crashing Container
k logs crash
k describe pod crash | grep -A3 'Last State'Expected:
Konfiguration fehlt
starte
Last State: Terminated
Reason: Error
Exit Code: 1
k logs shows stdout and stderr mixed together. Exit code
1 means: the program itself gave up – the cause is in its output.
For a container that is running again after a restart, there is
k logs crash --previous. In our lab (Kubernetes 1.37.2 with
containerd 2.4.2), however, the command responded with
unable to retrieve container logs for containerd://…. Then
you go one level deeper – to the node the pod is running on:
k get pod crash -o jsonpath='{.spec.nodeName}{"\n"}'On that node:
sudo sh -c 'ls /var/log/pods/tag15_crash_*/crash/'
sudo crictl ps -a --name crash
sudo crictl logs $(sudo crictl ps -a --name crash -q | head -1)Expected (abbreviated):
3.log
CONTAINER IMAGE CREATED STATE NAME ATTEMPT ...
eb36c57c3e879 30ecbe1509090 38 seconds ago Exited crash 3 ...
starte
Konfiguration fehlt
The file 3.log belongs to the fourth run (counting from
0). In this lab the node kept only the most recent exited container –
that is why --previous found nothing. Remember the path:
--previous first, otherwise crictl and
/var/log/pods on the node.
Save your progress? Sign up for free
Drill 3: Image and Memory
k describe pod pull | grep -E 'Failed' | tail -2
k describe pod hungry | grep -A3 'Last State'Expected (abbreviated):
Warning Failed 2s (x3 over 45s) kubelet spec.containers{pull}: Failed to pull
image
"nginx:1.299": rpc error: code = NotFound desc = failed to pull and unpack image ...
Warning Failed 2s (x3 over 45s) kubelet spec.containers{pull}: Error: ErrImagePull
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Exit code 137 = 128 + 9: the kernel terminated the process with SIGKILL because it wanted more than the 100Mi of the limit. Fixing it means: raise the limit or make the app use less memory.
k delete pod crash pull hungry --nowSave your progress? Sign up for free
Drill 4: A Service Without Endpoints
k create deployment shop --image=registry.k8s.io/e2e-test-images/agnhost:2.53 \
--replicas=2 -- /agnhost netexec --http-port=8080
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Service
metadata:
name: shop
spec:
selector:
app: webshop
ports:
- port: 80
targetPort: 8080
EOF
k run client --image=busybox:1.37 -- sleep 3600
k exec client -- wget -qO- -T3 shop/hostnameExpected:
wget: can't connect to remote host (10.105.116.134): Connection refused
command terminated with exit code 1
DNS works (the IP is resolved), but nobody answers:
k get endpointslices -l kubernetes.io/service-name=shop
k get svc shop -o jsonpath='{.spec.selector}{"\n"}'
k get pods --show-labels | grep shop | head -1Expected:
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
shop-d4xtk IPv4 <unset> <unset> 4s
{"app":"webshop"}
shop-d9fdd5c8b-c5lll 1/1 Running 0 7s app=shop,pod-template-hash=d9fdd5c8b
The selector looks for app=webshop, the pods are labeled
app=shop:
k set selector svc shop app=shop
k exec client -- wget -qO- -T3 shop/hostname; echoExpected: shop-d9fdd5c8b-c5lll
Save your progress? Sign up for free
Drill 5: Endpoints Present – and Still Dead
k patch svc shop -p '{"spec":{"ports":[{"port":80,"targetPort":9090}]}}'
k exec client -- wget -qO- -T3 shop/hostname
k get endpointslices -l kubernetes.io/service-name=shopExpected:
wget: can't connect to remote host (10.105.116.134): Connection refused
command terminated with exit code 1
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
shop-d4xtk IPv4 9090 192.168.189.96,192.168.235.177 7s
This time there are pods behind it, but on the wrong port – the
PORTS column gives it away. The container listens on
8080:
k patch svc shop -p '{"spec":{"ports":[{"port":80,"targetPort":8080}]}}'Save your progress? Sign up for free
Drill 6: No Name Resolution
k scale deployment coredns -n kube-system --replicas=0
k exec client -- wget -qO- -T3 shop/hostname
k exec client -- nslookup shop.tag15.svc.cluster.localExpected:
wget: bad address 'shop'
command terminated with exit code 1
nslookup: write to '10.96.0.10': Connection refused
command terminated with exit code 1
bad address is always DNS. Check the DNS service like
any other Service:
k get svc,endpointslices -n kube-system -l k8s-app=kube-dns
k get deployment coredns -n kube-systemExpected (abbreviated):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP
NAME ADDRESSTYPE PORTS ENDPOINTS
endpointslice.discovery.k8s.io/kube-dns-5xj56 IPv4 <unset> <unset>
NAME READY UP-TO-DATE AVAILABLE AGE
coredns 0/0 0 0 105m
k scale deployment coredns -n kube-system --replicas=2
k rollout status deployment coredns -n kube-system
k exec client -- wget -qO- -T3 shop/hostname; echoExpected: a pod name again.
Save your progress? Sign up for free
Drill 7: A Policy Nobody Knew About
cat <<'EOF' | k apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: lockdown
spec:
podSelector:
matchLabels:
app: shop
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
role: frontend
EOF
k exec client -- wget -qO- -T3 shop/hostnameExpected:
wget: download timed out
command terminated with exit code 1
timed out instead of refused: the packets
disappear. That smells like a NetworkPolicy:
k get netpol
k describe netpol lockdown | grep -A4 'Allowing ingress'Expected (abbreviated):
NAME POD-SELECTOR AGE
lockdown app=shop 4s
Allowing ingress traffic:
To Port: <any> (traffic allowed to all ports)
From:
PodSelector: role=frontend
The client is not a frontend. If it is allowed in, it gets the label:
k label pod client role=frontend
k exec client -- wget -qO- -T3 shop/hostname; echoExpected: a pod name again.
Save your progress? Sign up for free
Exercise: Speed Drills
An app with a sidecar – the native sidecar runs as an init container
with restartPolicy: Always:
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
name: app-with-sidecar
spec:
initContainers:
- name: log-shipper
image: busybox:1.37
restartPolicy: Always
command: ["sh", "-c", "tail -F /var/log/app/app.log"]
volumeMounts:
- name: logs
mountPath: /var/log/app
containers:
- name: app
image: busybox:1.37
command: ["sh", "-c", "i=0; while true; do i=$((i+1));
echo Bestellung $i >> /var/log/app/app.log; sleep 2; done"]
volumeMounts:
- name: logs
mountPath: /var/log/app
volumes:
- name: logs
emptyDir: {}
EOF- Show the last two lines of the
log-shippersidecar. Target time: 20 s - Show the last log line of each
shoppod, prefixed with the pod name. Target time: 30 s - Show all containers of
app-with-sidecar, only the last 4 seconds. Target time: 20 s
Solution
k logs app-with-sidecar -c log-shipper --tail=2
k logs -l app=shop --prefix --tail=1
k logs app-with-sidecar --all-containers --since=4sExpected (excerpt):
Bestellung 3
Bestellung 4
[pod/shop-d9fdd5c8b-c5lll/agnhost] I1117 22:57:25.389930 1 log.go:245] GET /hostname
[pod/shop-d9fdd5c8b-qqplx/agnhost] I1117 22:58:05.147661 1 log.go:245] GET /hostname
Bestellung 3
Bestellung 4
Without -c, k logs takes the first regular
container and tells you so:
Defaulted container "app" out of: app, log-shipper (init).
Save your progress? Sign up for free
Checkpoint
k get endpointslices -l kubernetes.io/service-name=shop \
-o jsonpath='{.items[0].ports[0].port}{"\n"}'Expected: 8080
k get deploy coredns -n kube-system -o jsonpath='{.status.readyReplicas}{"\n"}'Expected: 2
k get pod app-with-sidecarExpected: READY 2/2,
Running.
Challenge
The API is Running but never ready, and the Service does
not respond. Reproduce this and find the fault:
cat <<'EOF' | k apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 1
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: agnhost
image: registry.k8s.io/e2e-test-images/agnhost:2.53
command: ["/agnhost", "netexec", "--http-port=8080"]
readinessProbe:
httpGet:
path: /healthz
port: 8081
periodSeconds: 3
EOF
k expose deployment api --port=80 --target-port=8080Hint: An endpoint that is not ready
gets no traffic.
Solution
k get pods -l app=api
k get endpointslices -l kubernetes.io/service-name=api \
-o jsonpath='{.items[0].endpoints[0].conditions}{"\n"}'
k describe pod -l app=api | grep 'Readiness probe failed' | tail -1Expected (abbreviated):
NAME READY STATUS RESTARTS AGE
api-7bcb94444d-hnhwm 0/1 Running 0 26s
{"ready":false,"serving":false,"terminating":false}
Warning Unhealthy 1s (x10 over 26s) kubelet spec.containers{agnhost}: Readiness
probe
failed: Get "http://192.168.189.106:8081/healthz": dial tcp 192.168.189.106:8081:
connect:
connection refused
The probe queries port 8081, the app listens on 8080:
k patch deployment api --type=json -p='[{"op":"replace",
"path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":8080}]'
k rollout status deployment api
k exec client -- wget -qO- -T3 api/hostname; echoExpected:
deployment "api" successfully rolled out
api-697bfbbbcf-xqncz
Clean up:
k config set-context --current --namespace=default
k delete ns tag15Save your progress? Sign up for free
Exam Tip
- Read the error message carefully:
refused(nobody is listening),timed out(packets disappear),bad address(DNS) – three causes, three tools. k get endpointslicesanswers in one command whether selector, port, and readiness are correct.k describe podand the events below it first, logs afterwards.
Enterprise Note
- Logs belong in central collection (Loki, Elasticsearch); on the node they rotate away.
- Point readiness and liveness probes at dedicated endpoints, not at the app’s home page.
- For network faults, use flow logs (Calico Whisker, Cilium Hubble) instead of guessing.