Day 5: Crashes and Restarts
Goal: You tell process crashes apart from restarts by the kubelet and fix command, memory limit, probes and read-only filesystem. · Duration: approx. 75 min · Prerequisite: Day 1
Briefing
- A container restarts because the process ends (exit code, log)
or because the kubelet kills it (liveness, event
Killing). The events tell you which case it is. OOMKilledwith 137 means: the kernel killed within the container’s memory limit.- Readiness restarts nothing; it only takes the Pod out of the Service.
- The backoff between restarts doubles from 10 s up to at most 5 min.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Wrong command | Exit 127 | Log: not found |
| App error | Exit 1 | logs --previous |
| Memory | OOMKilled, 137 |
Last State, kernel log |
| Liveness too strict | Restarts, Killing |
Events Unhealthy |
| Wrong readiness | 0/1, no restarts |
Events, EndpointSlice |
Drill 1: Wrong command and the backoff
Break it:
# on cp1
mkdir -p ~/debug/tag05 && cd ~/debug/tag05
kubectl create namespace tag05
cat > tippfehler.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: tippfehler, namespace: tag05}
spec:
replicas: 1
selector: {matchLabels: {app: tippfehler}}
template:
metadata: {labels: {app: tippfehler}}
spec:
containers:
- name: app
image: busybox:1.37
command: [sh, -c, 'echo "starte"; slep infinity']
EOF
kubectl apply -f tippfehler.yamlSymptom: Watch restarts and wait times for two minutes:
# on cp1
for i in $(seq 1 12); do sleep 10
kubectl -n tag05 get pod -l app=tippfehler --no-headers | awk '{print $3, $4, $5, $6}'
done
kubectl -n tag05 get pod -l app=tippfehler \
-o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.message}{"\n"}' \
| cut -c1-40Expected (abridged):
CrashLoopBackOff 1 (8s ago)
Error 2 (18s ago)
Error 3 (36s ago)
CrashLoopBackOff 3 (38s ago)
Error 4 (49s ago)
Error 4 (81s ago)
back-off 1m20s restarting failed
Investigate:
# on cp1
kubectl -n tag05 logs deploy/tippfehler
kubectl -n tag05 get pod -l app=tippfehler -o jsonpath=\
'{.items[0].status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'Expected:
starte
sh: slep: not found
127
Cause: A typo in the command. Why? The wait grows 10 s, 20 s, 40 s, 80 s up to 5 min and only resets after the container has run without errors for ten minutes. So a fix does not take effect immediately while the Pod is stuck in a long backoff – a rollout creates a fresh Pod.
Fix:
# on cp1
kubectl -n tag05 patch deploy tippfehler --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/containers/0/command/2",
"value": "echo \"starte\"; sleep infinity"}]'
kubectl -n tag05 rollout status deploy/tippfehlerVerify:
# on cp1
kubectl -n tag05 logs deploy/tippfehlerExpected (abridged):
starte
Save your progress? Sign up for free
Drill 2: The application gives up
Break it:
# on cp1
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag05}
spec:
replicas: 1
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
containers:
- name: web
image: nginxinc/nginx-unprivileged:1.29-alpine
command: [nginx, -g, 'daemon off;', -c, /etc/nginx/conf.d/app.conf]
EOF
kubectl apply -f web.yamlSymptom and investigation:
# on cp1
kubectl -n tag05 get pods -l app=web
kubectl -n tag05 logs deploy/webExpected (line wrapped):
NAME READY STATUS RESTARTS AGE
web-7c8cd947d6-zstdf 0/1 CrashLoopBackOff 4 (55s ago) 2m26s
2026/11/20 07:25:01 [emerg] 1#1: open() "/etc/nginx/conf.d/app.conf" failed
(2: No such file or directory)
nginx: [emerg] open() "/etc/nginx/conf.d/app.conf" failed (2: No such file or directory)
Cause: The command points to a configuration that does not exist in the image; nginx exits with code 1. The file belongs in the Pod as a ConfigMap.
Fix:
# on cp1
cat > web-conf.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag05}
data:
app.conf: |
pid /tmp/nginx.pid;
events {}
http {
server {
listen 8080;
location / { return 200 "ok\n"; }
}
}
EOF
kubectl apply -f web-conf.yaml
kubectl -n tag05 patch deploy web -p '
spec:
template:
spec:
containers:
- name: web
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes:
- name: conf
configMap: {name: web-conf}'
kubectl -n tag05 rollout status deploy/webVerify:
# on cp1
kubectl -n tag05 exec deploy/web -- wget -qO- 127.0.0.1:8080Expected:
ok
Tip:
wget localhost:8080reportsConnection refusedhere: inside the Pod,localhostresolves to::1first, but nginx only listens on IPv4.
Why patch without --type=merge? The default
(strategic merge) merges lists such as containers by name.
A JSON merge patch would replace the whole list and fail on the missing
image.
Save your progress? Sign up for free
Drill 3: OOMKilled
Break it: A process loads 100 MB into memory; the limit is 64Mi.
# on cp1
cat > speicher.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: speicher, namespace: tag05}
spec:
replicas: 1
selector: {matchLabels: {app: speicher}}
template:
metadata: {labels: {app: speicher}}
spec:
containers:
- name: app
image: busybox:1.37
command:
- sh
- -c
- echo "lade Cache"; head -c 100m /dev/zero | tail >/dev/null; sleep infinity
resources:
requests: {cpu: 50m, memory: 64Mi}
limits: {memory: 64Mi}
EOF
kubectl apply -f speicher.yamlSymptom:
# on cp1
kubectl -n tag05 get pods -l app=speicher -o wide | cut -c1-88Expected:
NAME READY STATUS RESTARTS AGE IP NODE
speicher-7489b76474-qncx6 0/1 OOMKilled 1 (17s ago) 20s 10.42.2.16 worker2
Investigate:
# on cp1
kubectl -n tag05 describe pod -l app=speicher \
| sed -n '/Last State/,/Exit Code/p;/Limits/,/memory/p'Expected:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Limits:
memory: 64Mi
The proof comes from the kernel of the node in the NODE
column:
# on worker2
sudo journalctl -k --since -5min | grep 'Killed process' | cut -c1-88Expected:
Nov 20 07:27:20 worker2 kernel: Memory cgroup out of memory: Killed process 29137 (sh) t
Nov 20 07:27:20 worker2 kernel: Memory cgroup out of memory: Killed process 29157 (head)
Nov 20 07:27:20 worker2 kernel: Memory cgroup out of memory: Killed process 29158 (tail)
Cause: tail buffers everything up to
the first line ending – with /dev/zero that means 100 MB.
Why all three processes? Under cgroup v2 the kubelet sets
memory.oom.group; the kernel then kills the whole
container, not just the largest process.
Fix: Set the limit to what is needed plus headroom:
# on cp1
kubectl -n tag05 set resources deploy/speicher \
--requests=memory=160Mi --limits=memory=160Mi
kubectl -n tag05 rollout status deploy/speicherVerify:
# on cp1
kubectl -n tag05 get pods -l app=speicher
kubectl -n tag05 logs deploy/speicherExpected:
NAME READY STATUS RESTARTS AGE
speicher-67f57fff74-l4z8r 1/1 Running 0 42s
lade Cache
Save your progress? Sign up for free
Drill 4: Liveness too strict
Break it: The application needs 40 seconds to start; the liveness probe gives it 10 s plus three failed attempts at 5 s each.
# on cp1
cat > langsam.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: langsam, namespace: tag05}
spec:
replicas: 1
selector: {matchLabels: {app: langsam}}
template:
metadata: {labels: {app: langsam}}
spec:
terminationGracePeriodSeconds: 5
containers:
- name: app
image: busybox:1.37
command:
- sh
- -c
- |
echo "Lade Daten (dauert 40 Sekunden) ..."
sleep 40
mkdir -p /www && echo ok > /www/healthz
exec httpd -f -p 8080 -h /www
livenessProbe:
httpGet: {path: /healthz, port: 8080}
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
EOF
kubectl apply -f langsam.yamlSymptom (after a little over a minute):
# on cp1
kubectl -n tag05 get pods -l app=langsamExpected:
NAME READY STATUS RESTARTS AGE
langsam-6b8fb65674-95vvb 1/1 Running 2 (15s ago) 75s
1/1 and still restarts: without a readiness probe, the
container counts as ready immediately, even if it does not answer
yet.
Investigate:
# on cp1
kubectl -n tag05 events --for $(kubectl -n tag05 get pod -l app=langsam -o name) \
| grep -E 'Unhealthy|Killing' | cut -c1-88
kubectl -n tag05 logs deploy/langsam --previousExpected (lines wrapped):
21s (x2 over 51s) Normal Killing Pod/langsam-6b8fb65674-95vvb Container app
failed liveness probe, will be restarted
1s (x7 over 61s) Warning Unhealthy Pod/langsam-6b8fb65674-95vvb Liveness probe
failed: Get "http://10.42.2.19:8080/healthz": dial tcp ...
Lade Daten (dauert 40 Sekunden) ...
Cause: After 10 + 3 × 5 = 25 seconds the kubelet
kills the container, long before the 40-second startup time is over. The
log ends in the middle of startup, without an error message – the
process was killed from outside. sh as PID 1 ignores
SIGTERM, so after the 5 s grace period SIGKILL follows: exit code 137
without OOMKilled.
Fix: A startup probe protects the start; liveness only kicks in afterwards:
# on cp1
kubectl -n tag05 patch deploy langsam -p '
spec:
template:
spec:
containers:
- name: app
startupProbe:
httpGet: {path: /healthz, port: 8080}
periodSeconds: 5
failureThreshold: 18'
kubectl -n tag05 rollout status deploy/langsamVerify:
# on cp1
kubectl -n tag05 get pods -l app=langsamExpected (abridged):
NAME READY STATUS RESTARTS AGE
langsam-79b659cc47-d22tt 1/1 Running 0 46s
18 × 5 s = 90 s of startup time allowed; the rollout now waits until the application really answers.
Save your progress? Sign up for free
Drill 5: Readiness aimed at the wrong target
Break it:
# on cp1
cat > nieready.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: nieready, namespace: tag05}
spec:
replicas: 1
selector: {matchLabels: {app: nieready}}
template:
metadata: {labels: {app: nieready}}
spec:
containers:
- name: app
image: busybox:1.37
command: [sh, -c, 'mkdir -p /www && echo ok > /www/healthz &&
exec httpd -f -p 8080 -h /www']
readinessProbe:
httpGet: {path: /ready, port: 8080}
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata: {name: nieready, namespace: tag05}
spec:
selector: {app: nieready}
ports: [{port: 80, targetPort: 8080}]
EOF
kubectl apply -f nieready.yamlSymptom:
# on cp1
kubectl -n tag05 get pods -l app=niereadyExpected:
NAME READY STATUS RESTARTS AGE
nieready-85b946b75c-54r8c 0/1 Running 0 75s
Investigate:
# on cp1
kubectl -n tag05 events --for $(kubectl -n tag05 get pod -l app=nieready -o name) \
| grep Unhealthy | tail -1 | cut -c1-88
kubectl -n tag05 get endpointslices -l kubernetes.io/service-name=nieready
kubectl -n tag05 get endpointslices -l kubernetes.io/service-name=nieready \
-o jsonpath='{.items[0].endpoints[0].conditions}{"\n"}'Expected (line wrapped):
3s (x16 over 75s) Warning Unhealthy Pod/nieready-85b946b75c-54r8c Readiness
probe failed: HTTP probe failed with statuscode: 404
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
nieready-4j72l IPv4 8080 10.42.2.20 77s
{"ready":false,"serving":false,"terminating":false}
Cause: The probe asks for /ready, but
the application only knows /healthz – 404. Why is the IP
still in the EndpointSlice? Pods that are not ready are listed too, with
ready: false (in the old Endpoints object under
notReadyAddresses). kube-proxy sends no traffic to
them.
Fix:
# on cp1
kubectl -n tag05 patch deploy nieready --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path",
"value": "/healthz"}]'
kubectl -n tag05 rollout status deploy/niereadyVerify:
# on cp1
kubectl -n tag05 get endpointslices -l kubernetes.io/service-name=nieready \
-o jsonpath='{.items[0].endpoints[0].conditions}{"\n"}'Expected:
{"ready":true,"serving":true,"terminating":false}
Save your progress? Sign up for free
Drill 6: readOnlyRootFilesystem
Break it:
# on cp1
cat > webro.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: webro, namespace: tag05}
spec:
replicas: 1
selector: {matchLabels: {app: webro}}
template:
metadata: {labels: {app: webro}}
spec:
containers:
- name: web
image: nginxinc/nginx-unprivileged:1.29-alpine
securityContext: {readOnlyRootFilesystem: true}
EOF
kubectl apply -f webro.yamlSymptom and investigation:
# on cp1
kubectl -n tag05 get pods -l app=webro
kubectl -n tag05 logs deploy/webro | grep -iE 'read-only|can not modify' | cut -c1-88Expected (lines wrapped):
NAME READY STATUS RESTARTS AGE
webro-668d7bff5f-xg6r8 1/1 Running 2 (13s ago) 16s
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf
(read-only file system?)
2026/11/20 07:32:17 [emerg] 1#1: mkdir() "/tmp/proxy_temp" failed (30: Read-only
file system)
nginx: [emerg] mkdir() "/tmp/proxy_temp" failed (30: Read-only file system)
Cause: The root filesystem is read-only, but nginx
needs /tmp for temporary files. The first line is just an
info message from the entrypoint; the third is the actual abort.
Fix: Make writable only what the application really needs:
# on cp1
kubectl -n tag05 patch deploy webro -p '
spec:
template:
spec:
containers:
- name: web
volumeMounts: [{name: tmp, mountPath: /tmp}]
volumes:
- name: tmp
emptyDir: {sizeLimit: 64Mi}'
kubectl -n tag05 rollout status deploy/webroVerify:
# on cp1
kubectl -n tag05 exec deploy/webro -- wget -qO- 127.0.0.1:8080 | grep -o '<title>.*<'
kubectl -n tag05 exec deploy/webro -- touch /etc/nginx/testExpected:
<title>Welcome to nginx!<
touch: /etc/nginx/test: Read-only file system
command terminated with exit code 1
The protection stays in place for everything except
/tmp.
Save your progress? Sign up for free
Exercise: Speed Drills
- List the Pods in
tag05sorted by restarts. Target time: 30 s - Show the reason of the last terminated run for all Pods. Target time: 45 s
- Show the startup probe of
langsamas JSON. Target time: 30 s - Show the kernel’s OOM messages on
worker2. Target time: 20 s
Solution
# on cp1
kubectl -n tag05 get pods --sort-by='.status.containerStatuses[0].restartCount'
kubectl -n tag05 get pods -o custom-columns=NAME:.metadata.name,\
LAST:.status.containerStatuses[0].lastState.terminated.reason
kubectl -n tag05 get deploy langsam \
-o jsonpath='{.spec.template.spec.containers[0].startupProbe}{"\n"}'
ssh worker2 sudo journalctl -k | grep 'Killed process'Expected (abridged, all fixes applied):
NAME READY STATUS RESTARTS AGE
langsam-79b659cc47-d22tt 1/1 Running 0 3m8s
...
NAME LAST
langsam-79b659cc47-d22tt <none>
...
{"failureThreshold":18,"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},...}
Nov 20 07:27:20 worker2 kernel: Memory cgroup out of memory: Killed process 29158 ...
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag05 get deployExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
langsam 1/1 1 1 4m47s
nieready 1/1 1 1 4m47s
speicher 1/1 1 1 7m25s
tippfehler 1/1 1 1 10m
web 1/1 1 1 10m
webro 1/1 1 1 2m20s
Challenge
The Deployment waechter restarts every 15 to 20 seconds
at first, then with a growing backoff, even though httpd
runs fine. Find the cause and fix it.
# on cp1
cat > waechter.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: waechter, namespace: tag05}
spec:
replicas: 1
selector: {matchLabels: {app: waechter}}
template:
metadata: {labels: {app: waechter}}
spec:
terminationGracePeriodSeconds: 5
containers:
- name: app
image: busybox:1.37
command: [sh, -c, 'mkdir -p /www && echo ok > /www/index.html &&
exec httpd -f -p 8080 -h /www']
livenessProbe:
tcpSocket: {port: 80}
periodSeconds: 5
failureThreshold: 3
EOF
kubectl apply -f waechter.yamlHint: Who ends the container – the process itself or the kubelet?
Solution
# on cp1
kubectl -n tag05 events --for $(kubectl -n tag05 get pod -l app=waechter -o name) \
| grep -E 'Unhealthy|Killing' | cut -c1-88Expected (lines wrapped):
11s (x2 over 31s) Normal Killing Pod/waechter-5bf8fb5d6d-89w29 Container app
failed liveness probe, will be restarted
1s (x7 over 41s) Warning Unhealthy Pod/waechter-5bf8fb5d6d-89w29 Liveness probe
failed: dial tcp 10.42.2.23:80: ...
The probe checks port 80, but httpd listens on 8080.
# on cp1
kubectl -n tag05 patch deploy waechter --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/containers/0/livenessProbe/tcpSocket/port",
"value": 8080}]'
kubectl -n tag05 rollout status deploy/waechter
kubectl -n tag05 get pods -l app=waechterExpected (after 30 seconds):
NAME READY STATUS RESTARTS AGE
waechter-76495bb95b-z7h2b 1/1 Running 0 33s
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag05Expected:
namespace "tag05" deleted
Interview Note
- Liveness or readiness? Liveness restarts (process hangs), readiness removes from the Service (temporarily not ready). A liveness probe that depends on dependencies such as the database turns a DB outage into a restart storm.
- 137 – OOM or not?
Reason: OOMKilledor the kernel log; without either, it was a SIGKILL from outside (liveness, grace period expired). - Slow start?
startupProbeinstead of a hugeinitialDelaySeconds.
Enterprise Note
- Derive memory limits from measurements (
kubectl top, metrics over days), don’t guess; the JVM and friends need heap plus overhead. - The failure patterns here often only show up under load: probes with
timeoutSeconds: 1fail when the application is busy. readOnlyRootFilesystemas the default in Pod security policies, together with a list of the requiredemptyDirpaths per application.- Alerts on
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}and on rising restart counters are cheaper than any troubleshooting session.