Skip to content
Hands-on IT IT Drills

Day 5: Crashes and Restarts

Goal: You tell process crashes apart from restarts by the kubelet and fix command, memory limit, probes and read-only filesystem. · Duration: approx. 75 min · Prerequisite: Day 1

Briefing

  • A container restarts because the process ends (exit code, log) or because the kubelet kills it (liveness, event Killing). The events tell you which case it is.
  • OOMKilled with 137 means: the kernel killed within the container’s memory limit.
  • Readiness restarts nothing; it only takes the Pod out of the Service.
  • The backoff between restarts doubles from 10 s up to at most 5 min.
Fault Symptom Where the answer is
Wrong command Exit 127 Log: not found
App error Exit 1 logs --previous
Memory OOMKilled, 137 Last State, kernel log
Liveness too strict Restarts, Killing Events Unhealthy
Wrong readiness 0/1, no restarts Events, EndpointSlice

Drill 1: Wrong command and the backoff

Break it:

# on cp1
mkdir -p ~/debug/tag05 && cd ~/debug/tag05
kubectl create namespace tag05
cat > tippfehler.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: tippfehler, namespace: tag05}
spec:
  replicas: 1
  selector: {matchLabels: {app: tippfehler}}
  template:
    metadata: {labels: {app: tippfehler}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command: [sh, -c, 'echo "starte"; slep infinity']
EOF
kubectl apply -f tippfehler.yaml

Symptom: Watch restarts and wait times for two minutes:

# on cp1
for i in $(seq 1 12); do sleep 10
  kubectl -n tag05 get pod -l app=tippfehler --no-headers | awk '{print $3, $4, $5, $6}'
done
kubectl -n tag05 get pod -l app=tippfehler \
  -o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.message}{"\n"}' \
  | cut -c1-40

Expected (abridged):

CrashLoopBackOff 1 (8s ago)
Error 2 (18s ago)
Error 3 (36s ago)
CrashLoopBackOff 3 (38s ago)
Error 4 (49s ago)
Error 4 (81s ago)
back-off 1m20s restarting failed

Investigate:

# on cp1
kubectl -n tag05 logs deploy/tippfehler
kubectl -n tag05 get pod -l app=tippfehler -o jsonpath=\
'{.items[0].status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'

Expected:

starte
sh: slep: not found
127

Cause: A typo in the command. Why? The wait grows 10 s, 20 s, 40 s, 80 s up to 5 min and only resets after the container has run without errors for ten minutes. So a fix does not take effect immediately while the Pod is stuck in a long backoff – a rollout creates a fresh Pod.

Fix:

# on cp1
kubectl -n tag05 patch deploy tippfehler --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/containers/0/command/2",
  "value": "echo \"starte\"; sleep infinity"}]'
kubectl -n tag05 rollout status deploy/tippfehler

Verify:

# on cp1
kubectl -n tag05 logs deploy/tippfehler

Expected (abridged):

starte

Save your progress? Sign up for free

Drill 2: The application gives up

Break it:

# on cp1
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag05}
spec:
  replicas: 1
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: nginxinc/nginx-unprivileged:1.29-alpine
        command: [nginx, -g, 'daemon off;', -c, /etc/nginx/conf.d/app.conf]
EOF
kubectl apply -f web.yaml

Symptom and investigation:

# on cp1
kubectl -n tag05 get pods -l app=web
kubectl -n tag05 logs deploy/web

Expected (line wrapped):

NAME                   READY   STATUS             RESTARTS      AGE
web-7c8cd947d6-zstdf   0/1     CrashLoopBackOff   4 (55s ago)   2m26s
2026/11/20 07:25:01 [emerg] 1#1: open() "/etc/nginx/conf.d/app.conf" failed
  (2: No such file or directory)
nginx: [emerg] open() "/etc/nginx/conf.d/app.conf" failed (2: No such file or directory)

Cause: The command points to a configuration that does not exist in the image; nginx exits with code 1. The file belongs in the Pod as a ConfigMap.

Fix:

# on cp1
cat > web-conf.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag05}
data:
  app.conf: |
    pid /tmp/nginx.pid;
    events {}
    http {
      server {
        listen 8080;
        location / { return 200 "ok\n"; }
      }
    }
EOF
kubectl apply -f web-conf.yaml
kubectl -n tag05 patch deploy web -p '
spec:
  template:
    spec:
      containers:
      - name: web
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes:
      - name: conf
        configMap: {name: web-conf}'
kubectl -n tag05 rollout status deploy/web

Verify:

# on cp1
kubectl -n tag05 exec deploy/web -- wget -qO- 127.0.0.1:8080

Expected:

ok

Tip: wget localhost:8080 reports Connection refused here: inside the Pod, localhost resolves to ::1 first, but nginx only listens on IPv4.

Why patch without --type=merge? The default (strategic merge) merges lists such as containers by name. A JSON merge patch would replace the whole list and fail on the missing image.

Save your progress? Sign up for free

Drill 3: OOMKilled

Break it: A process loads 100 MB into memory; the limit is 64Mi.

# on cp1
cat > speicher.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: speicher, namespace: tag05}
spec:
  replicas: 1
  selector: {matchLabels: {app: speicher}}
  template:
    metadata: {labels: {app: speicher}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command:
        - sh
        - -c
        - echo "lade Cache"; head -c 100m /dev/zero | tail >/dev/null; sleep infinity
        resources:
          requests: {cpu: 50m, memory: 64Mi}
          limits: {memory: 64Mi}
EOF
kubectl apply -f speicher.yaml

Symptom:

# on cp1
kubectl -n tag05 get pods -l app=speicher -o wide | cut -c1-88

Expected:

NAME                        READY   STATUS      RESTARTS      AGE   IP           NODE
speicher-7489b76474-qncx6   0/1     OOMKilled   1 (17s ago)   20s   10.42.2.16   worker2

Investigate:

# on cp1
kubectl -n tag05 describe pod -l app=speicher \
  | sed -n '/Last State/,/Exit Code/p;/Limits/,/memory/p'

Expected:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Limits:
      memory:  64Mi

The proof comes from the kernel of the node in the NODE column:

# on worker2
sudo journalctl -k --since -5min | grep 'Killed process' | cut -c1-88

Expected:

Nov 20 07:27:20 worker2 kernel: Memory cgroup out of memory: Killed process 29137 (sh) t
Nov 20 07:27:20 worker2 kernel: Memory cgroup out of memory: Killed process 29157 (head)
Nov 20 07:27:20 worker2 kernel: Memory cgroup out of memory: Killed process 29158 (tail)

Cause: tail buffers everything up to the first line ending – with /dev/zero that means 100 MB. Why all three processes? Under cgroup v2 the kubelet sets memory.oom.group; the kernel then kills the whole container, not just the largest process.

Fix: Set the limit to what is needed plus headroom:

# on cp1
kubectl -n tag05 set resources deploy/speicher \
  --requests=memory=160Mi --limits=memory=160Mi
kubectl -n tag05 rollout status deploy/speicher

Verify:

# on cp1
kubectl -n tag05 get pods -l app=speicher
kubectl -n tag05 logs deploy/speicher

Expected:

NAME                        READY   STATUS    RESTARTS   AGE
speicher-67f57fff74-l4z8r   1/1     Running   0          42s
lade Cache

Save your progress? Sign up for free

Drill 4: Liveness too strict

Break it: The application needs 40 seconds to start; the liveness probe gives it 10 s plus three failed attempts at 5 s each.

# on cp1
cat > langsam.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: langsam, namespace: tag05}
spec:
  replicas: 1
  selector: {matchLabels: {app: langsam}}
  template:
    metadata: {labels: {app: langsam}}
    spec:
      terminationGracePeriodSeconds: 5
      containers:
      - name: app
        image: busybox:1.37
        command:
        - sh
        - -c
        - |
          echo "Lade Daten (dauert 40 Sekunden) ..."
          sleep 40
          mkdir -p /www && echo ok > /www/healthz
          exec httpd -f -p 8080 -h /www
        livenessProbe:
          httpGet: {path: /healthz, port: 8080}
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 3
EOF
kubectl apply -f langsam.yaml

Symptom (after a little over a minute):

# on cp1
kubectl -n tag05 get pods -l app=langsam

Expected:

NAME                       READY   STATUS    RESTARTS      AGE
langsam-6b8fb65674-95vvb   1/1     Running   2 (15s ago)   75s

1/1 and still restarts: without a readiness probe, the container counts as ready immediately, even if it does not answer yet.

Investigate:

# on cp1
kubectl -n tag05 events --for $(kubectl -n tag05 get pod -l app=langsam -o name) \
  | grep -E 'Unhealthy|Killing' | cut -c1-88
kubectl -n tag05 logs deploy/langsam --previous

Expected (lines wrapped):

21s (x2 over 51s)   Normal    Killing     Pod/langsam-6b8fb65674-95vvb   Container app
  failed liveness probe, will be restarted
1s (x7 over 61s)    Warning   Unhealthy   Pod/langsam-6b8fb65674-95vvb   Liveness probe
  failed: Get "http://10.42.2.19:8080/healthz": dial tcp ...
Lade Daten (dauert 40 Sekunden) ...

Cause: After 10 + 3 × 5 = 25 seconds the kubelet kills the container, long before the 40-second startup time is over. The log ends in the middle of startup, without an error message – the process was killed from outside. sh as PID 1 ignores SIGTERM, so after the 5 s grace period SIGKILL follows: exit code 137 without OOMKilled.

Fix: A startup probe protects the start; liveness only kicks in afterwards:

# on cp1
kubectl -n tag05 patch deploy langsam -p '
spec:
  template:
    spec:
      containers:
      - name: app
        startupProbe:
          httpGet: {path: /healthz, port: 8080}
          periodSeconds: 5
          failureThreshold: 18'
kubectl -n tag05 rollout status deploy/langsam

Verify:

# on cp1
kubectl -n tag05 get pods -l app=langsam

Expected (abridged):

NAME                       READY   STATUS    RESTARTS   AGE
langsam-79b659cc47-d22tt   1/1     Running   0          46s

18 × 5 s = 90 s of startup time allowed; the rollout now waits until the application really answers.

Save your progress? Sign up for free

Drill 5: Readiness aimed at the wrong target

Break it:

# on cp1
cat > nieready.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: nieready, namespace: tag05}
spec:
  replicas: 1
  selector: {matchLabels: {app: nieready}}
  template:
    metadata: {labels: {app: nieready}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command: [sh, -c, 'mkdir -p /www && echo ok > /www/healthz &&
          exec httpd -f -p 8080 -h /www']
        readinessProbe:
          httpGet: {path: /ready, port: 8080}
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata: {name: nieready, namespace: tag05}
spec:
  selector: {app: nieready}
  ports: [{port: 80, targetPort: 8080}]
EOF
kubectl apply -f nieready.yaml

Symptom:

# on cp1
kubectl -n tag05 get pods -l app=nieready

Expected:

NAME                        READY   STATUS    RESTARTS   AGE
nieready-85b946b75c-54r8c   0/1     Running   0          75s

Investigate:

# on cp1
kubectl -n tag05 events --for $(kubectl -n tag05 get pod -l app=nieready -o name) \
  | grep Unhealthy | tail -1 | cut -c1-88
kubectl -n tag05 get endpointslices -l kubernetes.io/service-name=nieready
kubectl -n tag05 get endpointslices -l kubernetes.io/service-name=nieready \
  -o jsonpath='{.items[0].endpoints[0].conditions}{"\n"}'

Expected (line wrapped):

3s (x16 over 75s)   Warning   Unhealthy   Pod/nieready-85b946b75c-54r8c   Readiness
  probe failed: HTTP probe failed with statuscode: 404
NAME             ADDRESSTYPE   PORTS   ENDPOINTS    AGE
nieready-4j72l   IPv4          8080    10.42.2.20   77s
{"ready":false,"serving":false,"terminating":false}

Cause: The probe asks for /ready, but the application only knows /healthz – 404. Why is the IP still in the EndpointSlice? Pods that are not ready are listed too, with ready: false (in the old Endpoints object under notReadyAddresses). kube-proxy sends no traffic to them.

Fix:

# on cp1
kubectl -n tag05 patch deploy nieready --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path",
  "value": "/healthz"}]'
kubectl -n tag05 rollout status deploy/nieready

Verify:

# on cp1
kubectl -n tag05 get endpointslices -l kubernetes.io/service-name=nieready \
  -o jsonpath='{.items[0].endpoints[0].conditions}{"\n"}'

Expected:

{"ready":true,"serving":true,"terminating":false}

Save your progress? Sign up for free

Drill 6: readOnlyRootFilesystem

Break it:

# on cp1
cat > webro.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: webro, namespace: tag05}
spec:
  replicas: 1
  selector: {matchLabels: {app: webro}}
  template:
    metadata: {labels: {app: webro}}
    spec:
      containers:
      - name: web
        image: nginxinc/nginx-unprivileged:1.29-alpine
        securityContext: {readOnlyRootFilesystem: true}
EOF
kubectl apply -f webro.yaml

Symptom and investigation:

# on cp1
kubectl -n tag05 get pods -l app=webro
kubectl -n tag05 logs deploy/webro | grep -iE 'read-only|can not modify' | cut -c1-88

Expected (lines wrapped):

NAME                     READY   STATUS    RESTARTS      AGE
webro-668d7bff5f-xg6r8   1/1     Running   2 (13s ago)   16s
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf
  (read-only file system?)
2026/11/20 07:32:17 [emerg] 1#1: mkdir() "/tmp/proxy_temp" failed (30: Read-only
  file system)
nginx: [emerg] mkdir() "/tmp/proxy_temp" failed (30: Read-only file system)

Cause: The root filesystem is read-only, but nginx needs /tmp for temporary files. The first line is just an info message from the entrypoint; the third is the actual abort.

Fix: Make writable only what the application really needs:

# on cp1
kubectl -n tag05 patch deploy webro -p '
spec:
  template:
    spec:
      containers:
      - name: web
        volumeMounts: [{name: tmp, mountPath: /tmp}]
      volumes:
      - name: tmp
        emptyDir: {sizeLimit: 64Mi}'
kubectl -n tag05 rollout status deploy/webro

Verify:

# on cp1
kubectl -n tag05 exec deploy/webro -- wget -qO- 127.0.0.1:8080 | grep -o '<title>.*<'
kubectl -n tag05 exec deploy/webro -- touch /etc/nginx/test

Expected:

<title>Welcome to nginx!<
touch: /etc/nginx/test: Read-only file system
command terminated with exit code 1

The protection stays in place for everything except /tmp.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. List the Pods in tag05 sorted by restarts. Target time: 30 s
  2. Show the reason of the last terminated run for all Pods. Target time: 45 s
  3. Show the startup probe of langsam as JSON. Target time: 30 s
  4. Show the kernel’s OOM messages on worker2. Target time: 20 s

Solution

# on cp1
kubectl -n tag05 get pods --sort-by='.status.containerStatuses[0].restartCount'
kubectl -n tag05 get pods -o custom-columns=NAME:.metadata.name,\
LAST:.status.containerStatuses[0].lastState.terminated.reason
kubectl -n tag05 get deploy langsam \
  -o jsonpath='{.spec.template.spec.containers[0].startupProbe}{"\n"}'
ssh worker2 sudo journalctl -k | grep 'Killed process'

Expected (abridged, all fixes applied):

NAME                          READY   STATUS    RESTARTS   AGE
langsam-79b659cc47-d22tt      1/1     Running   0          3m8s
...
NAME                          LAST
langsam-79b659cc47-d22tt      <none>
...
{"failureThreshold":18,"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},...}
Nov 20 07:27:20 worker2 kernel: Memory cgroup out of memory: Killed process 29158 ...

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag05 get deploy

Expected:

NAME         READY   UP-TO-DATE   AVAILABLE   AGE
langsam      1/1     1            1           4m47s
nieready     1/1     1            1           4m47s
speicher     1/1     1            1           7m25s
tippfehler   1/1     1            1           10m
web          1/1     1            1           10m
webro        1/1     1            1           2m20s

Challenge

The Deployment waechter restarts every 15 to 20 seconds at first, then with a growing backoff, even though httpd runs fine. Find the cause and fix it.

# on cp1
cat > waechter.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: waechter, namespace: tag05}
spec:
  replicas: 1
  selector: {matchLabels: {app: waechter}}
  template:
    metadata: {labels: {app: waechter}}
    spec:
      terminationGracePeriodSeconds: 5
      containers:
      - name: app
        image: busybox:1.37
        command: [sh, -c, 'mkdir -p /www && echo ok > /www/index.html &&
          exec httpd -f -p 8080 -h /www']
        livenessProbe:
          tcpSocket: {port: 80}
          periodSeconds: 5
          failureThreshold: 3
EOF
kubectl apply -f waechter.yaml

Hint: Who ends the container – the process itself or the kubelet?

Solution

# on cp1
kubectl -n tag05 events --for $(kubectl -n tag05 get pod -l app=waechter -o name) \
  | grep -E 'Unhealthy|Killing' | cut -c1-88

Expected (lines wrapped):

11s (x2 over 31s)   Normal    Killing     Pod/waechter-5bf8fb5d6d-89w29   Container app
  failed liveness probe, will be restarted
1s (x7 over 41s)    Warning   Unhealthy   Pod/waechter-5bf8fb5d6d-89w29   Liveness probe
  failed: dial tcp 10.42.2.23:80: ...

The probe checks port 80, but httpd listens on 8080.

# on cp1
kubectl -n tag05 patch deploy waechter --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/containers/0/livenessProbe/tcpSocket/port",
  "value": 8080}]'
kubectl -n tag05 rollout status deploy/waechter
kubectl -n tag05 get pods -l app=waechter

Expected (after 30 seconds):

NAME                        READY   STATUS    RESTARTS   AGE
waechter-76495bb95b-z7h2b   1/1     Running   0          33s

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag05

Expected:

namespace "tag05" deleted

Interview Note

  • Liveness or readiness? Liveness restarts (process hangs), readiness removes from the Service (temporarily not ready). A liveness probe that depends on dependencies such as the database turns a DB outage into a restart storm.
  • 137 – OOM or not? Reason: OOMKilled or the kernel log; without either, it was a SIGKILL from outside (liveness, grace period expired).
  • Slow start? startupProbe instead of a huge initialDelaySeconds.

Enterprise Note

  • Derive memory limits from measurements (kubectl top, metrics over days), don’t guess; the JVM and friends need heap plus overhead.
  • The failure patterns here often only show up under load: probes with timeoutSeconds: 1 fail when the application is busy.
  • readOnlyRootFilesystem as the default in Pod security policies, together with a list of the required emptyDir paths per application.
  • Alerts on kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} and on rising restart counters are cheaper than any troubleshooting session.