Skip to content
Hands-on IT IT Drills

Day 15: Troubleshooting II – Broken Apps and Networks

Goal: From status, events, and logs you can tell in seconds why an app is not running or not reachable – and you fix it. · Duration: approx. 75 min · Prerequisite: Day 9

Briefing

CKA: Troubleshooting (30 %) – “Manage and evaluate container output streams”, “Troubleshoot services and networking”.

Status / symptom Usual cause Tool
CrashLoopBackOff, Error Command, config, crash k logs
ImagePullBackOff Name, tag, registry access k describe pod
OOMKilled, exit 137 Memory limit too small k describe pod
Service: Connection refused Selector, targetPort, readiness k get endpointslices
bad address DNS Check CoreDNS
download timed out NetworkPolicy k get netpol
k create ns tag15
k config set-context --current --namespace=tag15

Drill 1: Three Broken Pods

k run crash --image=busybox:1.37 -- \
  sh -c 'echo starte; echo Konfiguration fehlt >&2; exit 1'
k run pull --image=nginx:1.299
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hungry
spec:
  containers:
  - name: app
    image: polinux/stress:1.0.4
    command: ["stress", "--vm", "1", "--vm-bytes", "200M", "--vm-hang", "1"]
    resources:
      limits:
        memory: 100Mi
EOF
k get pods

Expected (after about 45 s):

NAME     READY   STATUS             RESTARTS      AGE
crash    0/1     Error              3 (30s ago)   46s
hungry   0/1     OOMKilled          3 (26s ago)   45s
pull     0/1     ImagePullBackOff   0             46s

The status alternates between Error and CrashLoopBackOff – Kubernetes waits longer and longer between restarts (up to five minutes).

Save your progress? Sign up for free

Drill 2: Logs of a Crashing Container

k logs crash
k describe pod crash | grep -A3 'Last State'

Expected:

Konfiguration fehlt
starte
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1

k logs shows stdout and stderr mixed together. Exit code 1 means: the program itself gave up – the cause is in its output.

For a container that is running again after a restart, there is k logs crash --previous. In our lab (Kubernetes 1.37.2 with containerd 2.4.2), however, the command responded with unable to retrieve container logs for containerd://…. Then you go one level deeper – to the node the pod is running on:

k get pod crash -o jsonpath='{.spec.nodeName}{"\n"}'

On that node:

sudo sh -c 'ls /var/log/pods/tag15_crash_*/crash/'
sudo crictl ps -a --name crash
sudo crictl logs $(sudo crictl ps -a --name crash -q | head -1)

Expected (abbreviated):

3.log
CONTAINER       IMAGE           CREATED          STATE     NAME    ATTEMPT   ...
eb36c57c3e879   30ecbe1509090   38 seconds ago   Exited    crash   3         ...
starte
Konfiguration fehlt

The file 3.log belongs to the fourth run (counting from 0). In this lab the node kept only the most recent exited container – that is why --previous found nothing. Remember the path: --previous first, otherwise crictl and /var/log/pods on the node.

Save your progress? Sign up for free

Drill 3: Image and Memory

k describe pod pull | grep -E 'Failed' | tail -2
k describe pod hungry | grep -A3 'Last State'

Expected (abbreviated):

  Warning  Failed  2s (x3 over 45s)  kubelet  spec.containers{pull}: Failed to pull
image
"nginx:1.299": rpc error: code = NotFound desc = failed to pull and unpack image ...
  Warning  Failed  2s (x3 over 45s)  kubelet  spec.containers{pull}: Error: ErrImagePull
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Exit code 137 = 128 + 9: the kernel terminated the process with SIGKILL because it wanted more than the 100Mi of the limit. Fixing it means: raise the limit or make the app use less memory.

k delete pod crash pull hungry --now

Save your progress? Sign up for free

Drill 4: A Service Without Endpoints

k create deployment shop --image=registry.k8s.io/e2e-test-images/agnhost:2.53 \
  --replicas=2 -- /agnhost netexec --http-port=8080
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Service
metadata:
  name: shop
spec:
  selector:
    app: webshop
  ports:
  - port: 80
    targetPort: 8080
EOF
k run client --image=busybox:1.37 -- sleep 3600
k exec client -- wget -qO- -T3 shop/hostname

Expected:

wget: can't connect to remote host (10.105.116.134): Connection refused
command terminated with exit code 1

DNS works (the IP is resolved), but nobody answers:

k get endpointslices -l kubernetes.io/service-name=shop
k get svc shop -o jsonpath='{.spec.selector}{"\n"}'
k get pods --show-labels | grep shop | head -1

Expected:

NAME         ADDRESSTYPE   PORTS     ENDPOINTS   AGE
shop-d4xtk   IPv4          <unset>   <unset>     4s
{"app":"webshop"}
shop-d9fdd5c8b-c5lll   1/1   Running   0   7s   app=shop,pod-template-hash=d9fdd5c8b

The selector looks for app=webshop, the pods are labeled app=shop:

k set selector svc shop app=shop
k exec client -- wget -qO- -T3 shop/hostname; echo

Expected: shop-d9fdd5c8b-c5lll

Save your progress? Sign up for free

Drill 5: Endpoints Present – and Still Dead

k patch svc shop -p '{"spec":{"ports":[{"port":80,"targetPort":9090}]}}'
k exec client -- wget -qO- -T3 shop/hostname
k get endpointslices -l kubernetes.io/service-name=shop

Expected:

wget: can't connect to remote host (10.105.116.134): Connection refused
command terminated with exit code 1
NAME         ADDRESSTYPE   PORTS   ENDPOINTS                        AGE
shop-d4xtk   IPv4          9090    192.168.189.96,192.168.235.177   7s

This time there are pods behind it, but on the wrong port – the PORTS column gives it away. The container listens on 8080:

k patch svc shop -p '{"spec":{"ports":[{"port":80,"targetPort":8080}]}}'

Save your progress? Sign up for free

Drill 6: No Name Resolution

k scale deployment coredns -n kube-system --replicas=0
k exec client -- wget -qO- -T3 shop/hostname
k exec client -- nslookup shop.tag15.svc.cluster.local

Expected:

wget: bad address 'shop'
command terminated with exit code 1
nslookup: write to '10.96.0.10': Connection refused
command terminated with exit code 1

bad address is always DNS. Check the DNS service like any other Service:

k get svc,endpointslices -n kube-system -l k8s-app=kube-dns
k get deployment coredns -n kube-system

Expected (abbreviated):

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP
NAME                                            ADDRESSTYPE   PORTS     ENDPOINTS
endpointslice.discovery.k8s.io/kube-dns-5xj56   IPv4          <unset>   <unset>
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
coredns   0/0     0            0           105m
k scale deployment coredns -n kube-system --replicas=2
k rollout status deployment coredns -n kube-system
k exec client -- wget -qO- -T3 shop/hostname; echo

Expected: a pod name again.

Save your progress? Sign up for free

Drill 7: A Policy Nobody Knew About

cat <<'EOF' | k apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: lockdown
spec:
  podSelector:
    matchLabels:
      app: shop
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
EOF
k exec client -- wget -qO- -T3 shop/hostname

Expected:

wget: download timed out
command terminated with exit code 1

timed out instead of refused: the packets disappear. That smells like a NetworkPolicy:

k get netpol
k describe netpol lockdown | grep -A4 'Allowing ingress'

Expected (abbreviated):

NAME       POD-SELECTOR   AGE
lockdown   app=shop       4s
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From:
      PodSelector: role=frontend

The client is not a frontend. If it is allowed in, it gets the label:

k label pod client role=frontend
k exec client -- wget -qO- -T3 shop/hostname; echo

Expected: a pod name again.

Save your progress? Sign up for free

Exercise: Speed Drills

An app with a sidecar – the native sidecar runs as an init container with restartPolicy: Always:

cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: app-with-sidecar
spec:
  initContainers:
  - name: log-shipper
    image: busybox:1.37
    restartPolicy: Always
    command: ["sh", "-c", "tail -F /var/log/app/app.log"]
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  containers:
  - name: app
    image: busybox:1.37
    command: ["sh", "-c", "i=0; while true; do i=$((i+1));
      echo Bestellung $i >> /var/log/app/app.log; sleep 2; done"]
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  volumes:
  - name: logs
    emptyDir: {}
EOF
  1. Show the last two lines of the log-shipper sidecar. Target time: 20 s
  2. Show the last log line of each shop pod, prefixed with the pod name. Target time: 30 s
  3. Show all containers of app-with-sidecar, only the last 4 seconds. Target time: 20 s

Solution

k logs app-with-sidecar -c log-shipper --tail=2
k logs -l app=shop --prefix --tail=1
k logs app-with-sidecar --all-containers --since=4s

Expected (excerpt):

Bestellung 3
Bestellung 4
[pod/shop-d9fdd5c8b-c5lll/agnhost] I1117 22:57:25.389930  1 log.go:245] GET /hostname
[pod/shop-d9fdd5c8b-qqplx/agnhost] I1117 22:58:05.147661  1 log.go:245] GET /hostname
Bestellung 3
Bestellung 4

Without -c, k logs takes the first regular container and tells you so: Defaulted container "app" out of: app, log-shipper (init).

Save your progress? Sign up for free

Checkpoint

k get endpointslices -l kubernetes.io/service-name=shop \
  -o jsonpath='{.items[0].ports[0].port}{"\n"}'

Expected: 8080

k get deploy coredns -n kube-system -o jsonpath='{.status.readyReplicas}{"\n"}'

Expected: 2

k get pod app-with-sidecar

Expected: READY 2/2, Running.

Challenge

The API is Running but never ready, and the Service does not respond. Reproduce this and find the fault:

cat <<'EOF' | k apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: agnhost
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        command: ["/agnhost", "netexec", "--http-port=8080"]
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8081
          periodSeconds: 3
EOF
k expose deployment api --port=80 --target-port=8080

Hint: An endpoint that is not ready gets no traffic.

Solution

k get pods -l app=api
k get endpointslices -l kubernetes.io/service-name=api \
  -o jsonpath='{.items[0].endpoints[0].conditions}{"\n"}'
k describe pod -l app=api | grep 'Readiness probe failed' | tail -1

Expected (abbreviated):

NAME                   READY   STATUS    RESTARTS   AGE
api-7bcb94444d-hnhwm   0/1     Running   0          26s
{"ready":false,"serving":false,"terminating":false}
  Warning  Unhealthy  1s (x10 over 26s)  kubelet  spec.containers{agnhost}: Readiness
probe
failed: Get "http://192.168.189.106:8081/healthz": dial tcp 192.168.189.106:8081:
connect:
connection refused

The probe queries port 8081, the app listens on 8080:

k patch deployment api --type=json -p='[{"op":"replace",
  "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":8080}]'
k rollout status deployment api
k exec client -- wget -qO- -T3 api/hostname; echo

Expected:

deployment "api" successfully rolled out
api-697bfbbbcf-xqncz

Clean up:

k config set-context --current --namespace=default
k delete ns tag15

Save your progress? Sign up for free

Exam Tip

  • Read the error message carefully: refused (nobody is listening), timed out (packets disappear), bad address (DNS) – three causes, three tools.
  • k get endpointslices answers in one command whether selector, port, and readiness are correct.
  • k describe pod and the events below it first, logs afterwards.

Enterprise Note

  • Logs belong in central collection (Loki, Elasticsearch); on the node they rotate away.
  • Point readiness and liveness probes at dedicated endpoints, not at the app’s home page.
  • For network faults, use flow logs (Calico Whisker, Cilium Hubble) instead of guessing.