Skip to content
Hands-on IT IT Drills

Day 1: The Method

Goal: You narrow a fault down with fixed questions in a fixed order instead of guessing. · Duration: approx. 45 min · Prerequisite: Day 0

Briefing

  • Every fault in this book follows the same pattern: break it → symptom → narrow down → cause → fix → verify. Verification repeats the check from the symptom.
  • Narrowing down means working from the outside in; each question rules out one layer.
  • STATUS in kubectl get pods is a summary of phase, container state and reason – three different fields.
Layer Typical symptom Where the answer is
Scheduling Pending Events on the Pod (Day 3)
Image, runtime ImagePullBackOff Events, crictl (Day 4)
Workload CrashLoopBackOff Last State, logs (Day 5)
Network Timeout, refused EndpointSlice, DNS (Days 7–9)
Admission Forbidden, denied Message, events on the controller (Days 11–12)
Node, control plane NotReady, API hangs Journal, crictl (Days 14–16)

Drill 1: The Cluster First, Then the Pod

Before you examine a single Pod, you rule out the bottom layer: is the API server healthy, and are there warnings that have nothing to do with your problem?

# on cp1
mkdir -p ~/debug/tag01 && cd ~/debug/tag01
kubectl get --raw='/readyz?verbose' | grep -E 'etcd|passed'
kubectl events -A --types=Warning | tail -4 | cut -c1-88

Expected (abridged):

[+]etcd ok
[+]etcd-readiness ok
readyz check passed
kube-system          4m45s       Warning   NodeNotReady             Pod/kube-proxy-worke
kube-system          4m45s       Warning   NodeNotReady             Pod/rke2-canal-rwlsg
kube-system          4m45s       Warning   NodeNotReady             Pod/rke2-traefik-j8d
local-path-storage   4m45s       Warning   NodeNotReady             Pod/local-path-provi

The NodeNotReady warnings come from the Day 0 challenge. Why? By default, events live for one hour. What you see here is fresh – yesterday’s fault has no events left today.

Save your progress? Sign up for free

Drill 2: Break Something

An application expects the variable DB_HOST and aborts without it.

Break it:

# on cp1
kubectl create namespace tag01
cat > app.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: app, namespace: tag01}
spec:
  replicas: 1
  selector: {matchLabels: {app: app}}
  template:
    metadata: {labels: {app: app}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command:
        - sh
        - -c
        - |
          if [ -z "$DB_HOST" ]; then
            echo "FEHLER: DB_HOST ist nicht gesetzt" >&2
            exit 1
          fi
          echo "verbinde mit $DB_HOST"
          sleep 3600
EOF
kubectl apply -f app.yaml

Symptom (after about a minute):

# on cp1
kubectl -n tag01 get pods -o wide | cut -c1-88

Expected:

NAME                   READY   STATUS   RESTARTS      AGE   IP          NODE      NOMINA
app-7967b54879-q2hrx   0/1     Error    3 (33s ago)   50s   10.42.1.7   worker1   <none>

-o wide shows the node. If all broken Pods sit on the same node, the cause is more likely the node than the application. Here it is a single Pod.

Save your progress? Sign up for free

Drill 3: Phase, State and Reason

# on cp1
kubectl -n tag01 get pod -l app=app -o jsonpath='{.items[0].status.phase}{"\n"}'
kubectl -n tag01 get pod -l app=app \
  -o jsonpath='{.items[0].status.containerStatuses[0].state}{"\n"}' | cut -c1-88

Expected (two snapshots one minute apart):

Running
{"terminated":{"containerID":"containerd://8d2a14d0807af6a8a08a2d30394cbf15ba49cbee
Running
{"waiting":{"message":"back-off 1m20s restarting failed container=app pod=app-7967b5

Why? The phase stays Running as long as the Pod is assigned to a node and the kubelet restarts the container. The container alternates between terminated (right after the crash, STATUS shows Error) and waiting with reason CrashLoopBackOff. The delay doubles from 10 s up to a maximum of 5 min.

Save your progress? Sign up for free

Drill 4: describe and events

Investigate:

# on cp1
kubectl -n tag01 describe pod -l app=app | sed -n '/Last State:/,/Exit Code/p'
kubectl -n tag01 events --for $(kubectl -n tag01 get pod -l app=app -o name) \
  | cut -c1-88

Expected (abridged):

    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
LAST SEEN           TYPE      REASON      OBJECT                     MESSAGE
60s                 Normal    Scheduled   Pod/app-7967b54879-q2hrx   Successfully assign
21s (x4 over 59s)   Normal    Pulled      Pod/app-7967b54879-q2hrx   Container image "bu
21s (x4 over 59s)   Normal    Started     Pod/app-7967b54879-q2hrx   Container started
21s (x4 over 57s)   Warning   BackOff     Pod/app-7967b54879-q2hrx   Back-off restarting

Exit code 1, reason Error, no OOMKilled. The image is there, the container starts. That rules out scheduling, image and runtime: the process itself gives up. kubectl events sorts by time, newest at the bottom; --for filters on one object.

Save your progress? Sign up for free

Drill 5: Logs, Cause, Fix, Verify

# on cp1
kubectl -n tag01 logs deploy/app --previous
kubectl -n tag01 logs deploy/app

Expected while STATUS shows Error (line wrapped):

unable to retrieve container logs for
  containerd://09acf7e00de6e88b3580a62c2c24eadfdc29047d8264fa22736db6ffade8c97c
FEHLER: DB_HOST ist nicht gesetzt

Why? --previous means the run before the current container. If the current one is itself terminated, its log is the right one, and the one before has already been cleaned up. If the Pod is in CrashLoopBackOff, --previous returns the message. When in doubt, try both. (The message reads ‘ERROR: DB_HOST is not set’.)

Cause: The Deployment does not set DB_HOST, the application exits with code 1, and restartPolicy: Always keeps restarting it.

Fix:

# on cp1
kubectl -n tag01 set env deploy/app DB_HOST=db.tag01.svc.cluster.local
kubectl -n tag01 rollout status deploy/app

Verify:

# on cp1
kubectl -n tag01 get pods
kubectl -n tag01 logs deploy/app

Expected:

NAME                   READY   STATUS        RESTARTS      AGE
app-64cb96f58d-xp86h   1/1     Running       0             5s
app-7967b54879-q2hrx   0/1     Terminating   4 (66s ago)   2m30s
Found 2 pods, using pod/app-64cb96f58d-xp86h
verbinde mit db.tag01.svc.cluster.local

Tip: set env is the quick fix in the lab. In production the variable belongs in the manifest or the Git repo; otherwise the next deploy overwrites your fix.

Save your progress? Sign up for free

Drill 6: Reading Exit Codes

If a process ends because of a signal, the runtime reports 128 plus the signal number. With --restart=Never the container stays stopped after it ends, and you can read the code at leisure.

# on cp1
kubectl -n tag01 run code127 --image=busybox:1.37 --restart=Never -- sh -c 'slep 1'
kubectl -n tag01 run code126 --image=busybox:1.37 --restart=Never -- \
  sh -c '/etc/passwd'
kubectl -n tag01 run code128 --image=busybox:1.37 --restart=Never --command \
  -- /bin/slep
kubectl -n tag01 run code137 --image=busybox:1.37 --restart=Never -- \
  sh -c 'sleep 600 & kill -9 $!; wait $!'
kubectl -n tag01 get pods -o custom-columns=NAME:.metadata.name,\
PHASE:.status.phase,\
REASON:.status.containerStatuses[0].state.terminated.reason,\
EXIT:.status.containerStatuses[0].state.terminated.exitCode

Expected:

NAME                   PHASE     REASON       EXIT
app-64cb96f58d-xp86h   Running   <none>       <none>
code126                Failed    Error        126
code127                Failed    Error        127
code128                Failed    StartError   128
code137                Failed    Error        137
# on cp1
kubectl -n tag01 logs code127
kubectl -n tag01 logs code126
kubectl -n tag01 get pod code128 \
  -o jsonpath='{.status.containerStatuses[0].state.terminated.message}{"\n"}'

Expected (last line wrapped):

sh: slep: not found
sh: /etc/passwd: Permission denied
failed to create containerd task: failed to create shim task: OCI runtime create
  failed: runc create failed: unable to start container process: error during
  container init: exec: "/bin/slep": stat /bin/slep: no such file or directory
Code Meaning Typical cause
1 Application aborts Message in the log
126 / 127 not executable / not found Permissions, typo in command
128 + StartError Program missing, no shell in front Wrong path in command
137 / 143 SIGKILL / SIGTERM OOM or liveness (Day 5) / normal shutdown

Why? With --restart=Never the phase is Failed, not Running. A shell in front turns “program missing” into code 127; without a shell, runc already fails, and the kubelet reports StartError with 128.

Save your progress? Sign up for free

Drill 7: explain and auth can-i

# on cp1
kubectl explain pod.spec.restartPolicy | head -8
kubectl auth whoami | head -3 | cut -c1-88
kubectl auth can-i list pods -n tag01 --as=system:serviceaccount:tag01:default

Expected:

KIND:       Pod
VERSION:    v1

FIELD: restartPolicy <string>
ENUM:
    Always
    Never
    OnFailure
ATTRIBUTE                                           VALUE
Username                                            system:admin
Groups                                              [system:masters system:authenticated
no

explain reads the schema from the API server, matching the cluster version, and also works for CRDs. auth can-i --as answers permission questions without you having to log in as someone else (Day 11). You yourself are system:admin in system:masters – no RBAC stops you.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show only the warnings in tag01. Target time: 15 s
  2. Print the phase of the app Pod without typing its name. Target time: 30 s
  3. May the ServiceAccount default in tag01 read Secrets? Target time: 20 s
  4. Show the allowed values of pod.spec.containers.imagePullPolicy. Target time: 20 s

Solution

# on cp1
kubectl -n tag01 events --types=Warning
kubectl -n tag01 get pod -l app=app -o jsonpath='{.items[0].status.phase}{"\n"}'
kubectl auth can-i get secrets -n tag01 --as=system:serviceaccount:tag01:default
kubectl explain pod.spec.containers.imagePullPolicy

Expected (abridged):

LAST SEEN               TYPE      REASON    OBJECT                        MESSAGE
2m21s (x6 over 4m17s)   Warning   BackOff   Pod/app-7967b54879-q2hrx      Back-off resta
92s                     Warning   Failed    Pod/code128                   Error: failed
Running
no
FIELD: imagePullPolicy <string>
ENUM:
    Always
    IfNotPresent
    Never

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag01 get deploy app

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
app    1/1     1            1           4m
# on cp1
kubectl -n tag01 get pods -l app=app -o custom-columns=NAME:.metadata.name,\
RESTARTS:.status.containerStatuses[0].restartCount

Expected: one Pod with 0 restarts.

Challenge

Create a Deployment with kubectl -n tag01 create deployment rechner --image=busybox:1.37 -- sh -c 'echo $((100/0))'. With one kubectl get, find the restarts, reason and exit code of the last run of all Pods in the namespace, then the process’s message.

Hint: lastState.terminated sits next to state in the containerStatuses entry.

Solution

# on cp1
kubectl -n tag01 create deployment rechner --image=busybox:1.37 \
  -- sh -c 'echo $((100/0))'
kubectl -n tag01 get pods -o custom-columns=NAME:.metadata.name,\
RESTARTS:.status.containerStatuses[0].restartCount,\
REASON:.status.containerStatuses[0].lastState.terminated.reason,\
EXIT:.status.containerStatuses[0].lastState.terminated.exitCode
kubectl -n tag01 logs deploy/rechner

Expected (abridged, without the code… Pods from Drill 6):

NAME                      RESTARTS   REASON   EXIT
app-64cb96f58d-xp86h      0          <none>   <none>
rechner-6d689cd4f-4bzk2   3          Error    2
sh: divide by zero

Exit code 2 is set by the shell itself here. It also belongs to the group “application aborts, answer in the log”.

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag01
kubectl get namespace tag01

Expected:

namespace "tag01" deleted
Error from server (NotFound): namespaces "tag01" not found

Interview Note

  • A Pod is in CrashLoopBackOff – what do you do? describe for Last State and the exit code, then logs --previous (or without it if the container is currently terminated). Never delete the Pod first.
  • What does exit code 137 mean? SIGKILL; with reason OOMKilled it is the memory limit, without it usually a kill after a failed liveness probe.
  • Why is a crashing Pod Running? The phase describes the Pod, not the container; it is assigned and gets restarted.

Enterprise Note

  • Events disappear after one hour. An event exporter or central logging keeps them for post-incident analysis.
  • logs --previous keeps exactly one old run. For more, you need log collection on the nodes (Fluent Bit, Vector, Promtail’s successor).
  • Write down what you checked during every troubleshooting session – that way the ticket is already half documented.
  • Fixes via kubectl set/edit are emergency measures; otherwise GitOps tools revert them at the next sync.