Day 1: The Method
Goal: You narrow a fault down with fixed questions in a fixed order instead of guessing. · Duration: approx. 45 min · Prerequisite: Day 0
Briefing
- Every fault in this book follows the same pattern: break it → symptom → narrow down → cause → fix → verify. Verification repeats the check from the symptom.
- Narrowing down means working from the outside in; each question rules out one layer.
STATUSinkubectl get podsis a summary of phase, container state and reason – three different fields.
| Layer | Typical symptom | Where the answer is |
|---|---|---|
| Scheduling | Pending |
Events on the Pod (Day 3) |
| Image, runtime | ImagePullBackOff |
Events, crictl (Day 4) |
| Workload | CrashLoopBackOff |
Last State, logs (Day 5) |
| Network | Timeout, refused |
EndpointSlice, DNS (Days 7–9) |
| Admission | Forbidden, denied |
Message, events on the controller (Days 11–12) |
| Node, control plane | NotReady, API hangs |
Journal, crictl (Days 14–16) |
Drill 1: The Cluster First, Then the Pod
Before you examine a single Pod, you rule out the bottom layer: is the API server healthy, and are there warnings that have nothing to do with your problem?
# on cp1
mkdir -p ~/debug/tag01 && cd ~/debug/tag01
kubectl get --raw='/readyz?verbose' | grep -E 'etcd|passed'
kubectl events -A --types=Warning | tail -4 | cut -c1-88Expected (abridged):
[+]etcd ok
[+]etcd-readiness ok
readyz check passed
kube-system 4m45s Warning NodeNotReady Pod/kube-proxy-worke
kube-system 4m45s Warning NodeNotReady Pod/rke2-canal-rwlsg
kube-system 4m45s Warning NodeNotReady Pod/rke2-traefik-j8d
local-path-storage 4m45s Warning NodeNotReady Pod/local-path-provi
The NodeNotReady warnings come from the Day 0 challenge.
Why? By default, events live for one hour. What you see here is fresh –
yesterday’s fault has no events left today.
Save your progress? Sign up for free
Drill 2: Break Something
An application expects the variable DB_HOST and aborts
without it.
Break it:
# on cp1
kubectl create namespace tag01
cat > app.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: app, namespace: tag01}
spec:
replicas: 1
selector: {matchLabels: {app: app}}
template:
metadata: {labels: {app: app}}
spec:
containers:
- name: app
image: busybox:1.37
command:
- sh
- -c
- |
if [ -z "$DB_HOST" ]; then
echo "FEHLER: DB_HOST ist nicht gesetzt" >&2
exit 1
fi
echo "verbinde mit $DB_HOST"
sleep 3600
EOF
kubectl apply -f app.yamlSymptom (after about a minute):
# on cp1
kubectl -n tag01 get pods -o wide | cut -c1-88Expected:
NAME READY STATUS RESTARTS AGE IP NODE NOMINA
app-7967b54879-q2hrx 0/1 Error 3 (33s ago) 50s 10.42.1.7 worker1 <none>
-o wide shows the node. If all broken Pods sit on the
same node, the cause is more likely the node than the application. Here
it is a single Pod.
Save your progress? Sign up for free
Drill 3: Phase, State and Reason
# on cp1
kubectl -n tag01 get pod -l app=app -o jsonpath='{.items[0].status.phase}{"\n"}'
kubectl -n tag01 get pod -l app=app \
-o jsonpath='{.items[0].status.containerStatuses[0].state}{"\n"}' | cut -c1-88Expected (two snapshots one minute apart):
Running
{"terminated":{"containerID":"containerd://8d2a14d0807af6a8a08a2d30394cbf15ba49cbee
Running
{"waiting":{"message":"back-off 1m20s restarting failed container=app pod=app-7967b5
Why? The phase stays Running as long as the Pod is
assigned to a node and the kubelet restarts the container. The container
alternates between terminated (right after the crash,
STATUS shows Error) and waiting
with reason CrashLoopBackOff. The delay doubles from 10 s
up to a maximum of 5 min.
Save your progress? Sign up for free
Drill 4: describe and events
Investigate:
# on cp1
kubectl -n tag01 describe pod -l app=app | sed -n '/Last State:/,/Exit Code/p'
kubectl -n tag01 events --for $(kubectl -n tag01 get pod -l app=app -o name) \
| cut -c1-88Expected (abridged):
Last State: Terminated
Reason: Error
Exit Code: 1
LAST SEEN TYPE REASON OBJECT MESSAGE
60s Normal Scheduled Pod/app-7967b54879-q2hrx Successfully assign
21s (x4 over 59s) Normal Pulled Pod/app-7967b54879-q2hrx Container image "bu
21s (x4 over 59s) Normal Started Pod/app-7967b54879-q2hrx Container started
21s (x4 over 57s) Warning BackOff Pod/app-7967b54879-q2hrx Back-off restarting
Exit code 1, reason Error, no
OOMKilled. The image is there, the container starts. That
rules out scheduling, image and runtime: the process itself gives up.
kubectl events sorts by time, newest at the bottom;
--for filters on one object.
Save your progress? Sign up for free
Drill 5: Logs, Cause, Fix, Verify
# on cp1
kubectl -n tag01 logs deploy/app --previous
kubectl -n tag01 logs deploy/appExpected while STATUS shows
Error (line wrapped):
unable to retrieve container logs for
containerd://09acf7e00de6e88b3580a62c2c24eadfdc29047d8264fa22736db6ffade8c97c
FEHLER: DB_HOST ist nicht gesetzt
Why? --previous means the run before
the current container. If the current one is itself
terminated, its log is the right one, and the one before
has already been cleaned up. If the Pod is in
CrashLoopBackOff, --previous returns the
message. When in doubt, try both. (The message reads ‘ERROR: DB_HOST is
not set’.)
Cause: The Deployment does not set
DB_HOST, the application exits with code 1, and
restartPolicy: Always keeps restarting it.
Fix:
# on cp1
kubectl -n tag01 set env deploy/app DB_HOST=db.tag01.svc.cluster.local
kubectl -n tag01 rollout status deploy/appVerify:
# on cp1
kubectl -n tag01 get pods
kubectl -n tag01 logs deploy/appExpected:
NAME READY STATUS RESTARTS AGE
app-64cb96f58d-xp86h 1/1 Running 0 5s
app-7967b54879-q2hrx 0/1 Terminating 4 (66s ago) 2m30s
Found 2 pods, using pod/app-64cb96f58d-xp86h
verbinde mit db.tag01.svc.cluster.local
Tip:
set envis the quick fix in the lab. In production the variable belongs in the manifest or the Git repo; otherwise the next deploy overwrites your fix.
Save your progress? Sign up for free
Drill 6: Reading Exit Codes
If a process ends because of a signal, the runtime reports 128 plus
the signal number. With --restart=Never the container stays
stopped after it ends, and you can read the code at leisure.
# on cp1
kubectl -n tag01 run code127 --image=busybox:1.37 --restart=Never -- sh -c 'slep 1'
kubectl -n tag01 run code126 --image=busybox:1.37 --restart=Never -- \
sh -c '/etc/passwd'
kubectl -n tag01 run code128 --image=busybox:1.37 --restart=Never --command \
-- /bin/slep
kubectl -n tag01 run code137 --image=busybox:1.37 --restart=Never -- \
sh -c 'sleep 600 & kill -9 $!; wait $!'
kubectl -n tag01 get pods -o custom-columns=NAME:.metadata.name,\
PHASE:.status.phase,\
REASON:.status.containerStatuses[0].state.terminated.reason,\
EXIT:.status.containerStatuses[0].state.terminated.exitCodeExpected:
NAME PHASE REASON EXIT
app-64cb96f58d-xp86h Running <none> <none>
code126 Failed Error 126
code127 Failed Error 127
code128 Failed StartError 128
code137 Failed Error 137
# on cp1
kubectl -n tag01 logs code127
kubectl -n tag01 logs code126
kubectl -n tag01 get pod code128 \
-o jsonpath='{.status.containerStatuses[0].state.terminated.message}{"\n"}'Expected (last line wrapped):
sh: slep: not found
sh: /etc/passwd: Permission denied
failed to create containerd task: failed to create shim task: OCI runtime create
failed: runc create failed: unable to start container process: error during
container init: exec: "/bin/slep": stat /bin/slep: no such file or directory
| Code | Meaning | Typical cause |
|---|---|---|
| 1 | Application aborts | Message in the log |
| 126 / 127 | not executable / not found | Permissions, typo in command |
128 + StartError |
Program missing, no shell in front | Wrong path in command |
| 137 / 143 | SIGKILL / SIGTERM | OOM or liveness (Day 5) / normal shutdown |
Why? With --restart=Never the phase is
Failed, not Running. A shell in front turns
“program missing” into code 127; without a shell, runc already fails,
and the kubelet reports StartError with 128.
Save your progress? Sign up for free
Drill 7: explain and auth can-i
# on cp1
kubectl explain pod.spec.restartPolicy | head -8
kubectl auth whoami | head -3 | cut -c1-88
kubectl auth can-i list pods -n tag01 --as=system:serviceaccount:tag01:defaultExpected:
KIND: Pod
VERSION: v1
FIELD: restartPolicy <string>
ENUM:
Always
Never
OnFailure
ATTRIBUTE VALUE
Username system:admin
Groups [system:masters system:authenticated
no
explain reads the schema from the API server, matching
the cluster version, and also works for CRDs.
auth can-i --as answers permission questions without you
having to log in as someone else (Day 11). You yourself are
system:admin in system:masters – no RBAC stops
you.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show only the warnings in
tag01. Target time: 15 s - Print the phase of the
appPod without typing its name. Target time: 30 s - May the ServiceAccount
defaultintag01read Secrets? Target time: 20 s - Show the allowed values of
pod.spec.containers.imagePullPolicy. Target time: 20 s
Solution
# on cp1
kubectl -n tag01 events --types=Warning
kubectl -n tag01 get pod -l app=app -o jsonpath='{.items[0].status.phase}{"\n"}'
kubectl auth can-i get secrets -n tag01 --as=system:serviceaccount:tag01:default
kubectl explain pod.spec.containers.imagePullPolicyExpected (abridged):
LAST SEEN TYPE REASON OBJECT MESSAGE
2m21s (x6 over 4m17s) Warning BackOff Pod/app-7967b54879-q2hrx Back-off resta
92s Warning Failed Pod/code128 Error: failed
Running
no
FIELD: imagePullPolicy <string>
ENUM:
Always
IfNotPresent
Never
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag01 get deploy appExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
app 1/1 1 1 4m
# on cp1
kubectl -n tag01 get pods -l app=app -o custom-columns=NAME:.metadata.name,\
RESTARTS:.status.containerStatuses[0].restartCountExpected: one Pod with 0 restarts.
Challenge
Create a Deployment with
kubectl -n tag01 create deployment rechner --image=busybox:1.37 -- sh -c 'echo $((100/0))'.
With one kubectl get, find the restarts,
reason and exit code of the last run of all Pods in the namespace, then
the process’s message.
Hint: lastState.terminated sits next to
state in the containerStatuses entry.
Solution
# on cp1
kubectl -n tag01 create deployment rechner --image=busybox:1.37 \
-- sh -c 'echo $((100/0))'
kubectl -n tag01 get pods -o custom-columns=NAME:.metadata.name,\
RESTARTS:.status.containerStatuses[0].restartCount,\
REASON:.status.containerStatuses[0].lastState.terminated.reason,\
EXIT:.status.containerStatuses[0].lastState.terminated.exitCode
kubectl -n tag01 logs deploy/rechnerExpected (abridged, without the code…
Pods from Drill 6):
NAME RESTARTS REASON EXIT
app-64cb96f58d-xp86h 0 <none> <none>
rechner-6d689cd4f-4bzk2 3 Error 2
sh: divide by zero
Exit code 2 is set by the shell itself here. It also belongs to the group “application aborts, answer in the log”.
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag01
kubectl get namespace tag01Expected:
namespace "tag01" deleted
Error from server (NotFound): namespaces "tag01" not found
Interview Note
- A Pod is in
CrashLoopBackOff– what do you do?describeforLast Stateand the exit code, thenlogs --previous(or without it if the container is currentlyterminated). Never delete the Pod first. - What does exit code 137 mean? SIGKILL; with reason
OOMKilledit is the memory limit, without it usually a kill after a failed liveness probe. - Why is a crashing Pod
Running? The phase describes the Pod, not the container; it is assigned and gets restarted.
Enterprise Note
- Events disappear after one hour. An event exporter or central logging keeps them for post-incident analysis.
logs --previouskeeps exactly one old run. For more, you need log collection on the nodes (Fluent Bit, Vector, Promtail’s successor).- Write down what you checked during every troubleshooting session – that way the ticket is already half documented.
- Fixes via
kubectl set/editare emergency measures; otherwise GitOps tools revert them at the next sync.