Skip to content
Hands-on IT IT Drills

Day 4: Images and Container Start

Goal: You tell pull errors apart by their message, check images directly on the node and find errors that happen between pull and start. · Duration: approx. 60 min · Prerequisite: Day 1

Briefing

  • After scheduling, the kubelet takes over: pull the image (containerd), assemble the container from env, Secrets and volumes, start the process. Each phase has its own failure pattern.
  • ErrImagePull is the first failure, ImagePullBackOff is the wait until the next attempt. The cause is in the message behind it, not in the status.
  • exec format error is a start error, not a pull error: the image is there, but the kernel cannot execute the file.
Fault Symptom Where the answer is
Wrong tag ErrImagePull Message: not found
Registry unreachable ErrImagePull Message: no such host, HTTPS client
Private without Secret ImagePullBackOff Message: pull access denied
ConfigMap/Secret missing CreateContainerConfigError Event Failed on the Pod
Shebang, architecture CrashLoopBackOff Log: exec format error

Drill 1: Three pull errors, three messages

Break it:

# on cp1
mkdir -p ~/debug/tag04 && cd ~/debug/tag04
kubectl create namespace tag04
kubectl -n tag04 create deployment tippfehler \
  --image=nginxinc/nginx-unprivileged:1.29-alpne
kubectl -n tag04 create deployment unerreichbar --image=registry.tag04.invalid/web:1.0
kubectl -n tag04 create deployment privat --image=docker.io/dbglabor/privat-app:1.0

Symptom:

# on cp1
kubectl -n tag04 get pods

Expected:

NAME                            READY   STATUS             RESTARTS   AGE
privat-6b74c95474-p5shr         0/1     ImagePullBackOff   0          40s
tippfehler-7c975b5497-5cc46     0/1     ErrImagePull       0          40s
unerreichbar-699cd56b54-lmwrx   0/1     ErrImagePull       0          40s

Investigate: The message is in state.waiting.message (or under Events in describe):

# on cp1
for a in tippfehler unerreichbar privat; do
  kubectl -n tag04 get pod -l app=$a \
    -o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.message}{"\n"}'
done

Expected (lines wrapped, abridged):

rpc error: code = NotFound desc = failed to pull and unpack image
  "docker.io/nginxinc/nginx-unprivileged:1.29-alpne": ...
  docker.io/nginxinc/nginx-unprivileged:1.29-alpne: not found
failed to pull and unpack image "registry.tag04.invalid/web:1.0": ... failed to do
  request: Head "https://registry.tag04.invalid/v2/web/manifests/1.0": dial tcp:
  lookup registry.tag04.invalid on 127.0.0.53:53: no such host
Back-off pulling image "docker.io/dbglabor/privat-app:1.0": ErrImagePull: ... pull
  access denied, repository does not exist or may require authorization: server
  message: insufficient_scope: authorization failed

Cause: not found – the registry answers but does not know the tag. no such host – the node (not the Pod!) cannot resolve the name; the address 127.0.0.53 is the host’s systemd-resolved. pull access denied – the repository does not exist or it is private; Docker Hub deliberately does not tell the two apart.

Fix for the typo (with create deployment, the container name comes from the image name):

# on cp1
kubectl -n tag04 set image deploy/tippfehler \
  nginx-unprivileged=nginxinc/nginx-unprivileged:1.29-alpine
kubectl -n tag04 rollout status deploy/tippfehler
kubectl -n tag04 delete deploy unerreichbar privat

Verify:

# on cp1
kubectl -n tag04 get pods -l app=tippfehler

Expected: one Pod 1/1 Running. You rebuild the private case properly in Drill 4.

Save your progress? Sign up for free

Drill 2: Check on the node

You check whether a pull works on the node itself with crictl, without the kubelet. That separates “node cannot reach the registry” from “Pod spec is wrong”.

# on worker1
sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpne 2>&1 | tail -1 | cut -c1-88
sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpine
sudo -i crictl images | grep -E 'IMAGE|nginx'

Expected (abridged):

time="2026-11-20T07:09:08Z" level=error msg="pulling image: rpc error: code = NotFound d
Image is up to date for sha256:7312fa98595737958df41e20f21a2880dd9775f455c6bf76e638024dc
IMAGE                                   TAG                    IMAGE ID          ...
docker.io/nginxinc/nginx-unprivileged   1.29                   b3c400d96d1d1     ...
docker.io/nginxinc/nginx-unprivileged   1.29-alpine            7312fa9859573     ...

crictl reports Image is up to date after every successful pull. containerd expands short names like nginxinc/… to docker.io/….

Talos: There the commands are talosctl image pull and talosctl image list (Day 18).

Save your progress? Sign up for free

Drill 3: registries.yaml – mirror and HTTP registry

Docker Hub limits anonymous pulls per IP. A mirror is not configured in the Pod but in containerd on every node. RKE2 reads /etc/rancher/rke2/registries.yaml for this – only at startup. The second entry prepares Drill 4: the lab registry there only speaks HTTP, but containerd tries HTTPS for every registry unless told otherwise.

# on cp1, worker1 and worker2
sudo tee /etc/rancher/rke2/registries.yaml >/dev/null <<'EOF'
mirrors:
  docker.io:
    endpoint: ["https://mirror.gcr.io"]
  "172.16.0.126:30500":
    endpoint: ["http://172.16.0.126:30500"]
EOF
sudo chmod 600 /etc/rancher/rke2/registries.yaml

Warning: Restarting the RKE2 service restarts the kubelet and containerd. Running Pods survive it, but on cp1 the API is gone for a moment.

# on cp1
sudo systemctl restart rke2-server
# on worker1 and worker2
sudo systemctl restart rke2-agent

From this, RKE2 generates a hosts.toml per registry for containerd:

# on worker1
sudo cat /var/lib/rancher/rke2/agent/etc/containerd/certs.d/docker.io/hosts.toml

Expected:

# File generated by rke2. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]

[host]

[host."https://mirror.gcr.io/v2"]
  capabilities = ["pull", "resolve"]
# on worker2
sudo -i crictl pull busybox:1.36

Expected:

Image is up to date for sha256:b116e155074440ffd9e449559433feb4cd2341eb3554b1da1c638c

Why? containerd asks the mirror first and otherwise falls back to server. A broken mirror therefore does not show up right away: pulls just get slower and count against the limit again.

Save your progress? Sign up for free

Drill 4: Private registry and imagePullSecret

For the real path you need a registry with a password. You start registry:3 with an htpasswd file in the cluster, reachable on NodePort 30500.

# on cp1
kubectl -n tag04 run htpasswd --rm -i --quiet --restart=Never \
  --image=httpd:2.4-alpine -- htpasswd -Bbn labor labor-pw-123 > htpasswd
kubectl -n tag04 create secret generic registry-auth --from-file=htpasswd
cat > registry.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: registry, namespace: tag04}
spec:
  selector: {matchLabels: {app: registry}}
  template:
    metadata: {labels: {app: registry}}
    spec:
      containers:
      - name: registry
        image: registry:3.1.2
        env:
        - {name: REGISTRY_AUTH, value: htpasswd}
        - {name: REGISTRY_AUTH_HTPASSWD_REALM, value: labor}
        - {name: REGISTRY_AUTH_HTPASSWD_PATH, value: /auth/htpasswd}
        ports: [{containerPort: 5000}]
        volumeMounts: [{name: auth, mountPath: /auth}]
      volumes:
      - name: auth
        secret: {secretName: registry-auth}
---
apiVersion: v1
kind: Service
metadata: {name: registry, namespace: tag04}
spec:
  type: NodePort
  selector: {app: registry}
  ports: [{port: 5000, nodePort: 30500}]
EOF
kubectl apply -f registry.yaml
kubectl -n tag04 rollout status deploy/registry
curl -s -i http://172.16.0.126:30500/v2/ | grep -E 'HTTP|Authenticate'

Expected:

HTTP/1.1 401 Unauthorized
Www-Authenticate: Basic realm="labor"

Copy an image into it, using regctl in a short-lived Pod:

# on cp1
kubectl -n tag04 run regctl --restart=Never --image=regclient/regctl:v0.11.7-alpine \
  --command -- sh -c '
regctl registry set --tls disabled registry.tag04:5000 2>/dev/null
regctl registry login registry.tag04:5000 -u labor -p labor-pw-123 &&
regctl image copy traefik/whoami:v1.11 registry.tag04:5000/privat/whoami:1.0 &&
regctl tag ls registry.tag04:5000/privat/whoami'
kubectl -n tag04 wait pod/regctl --for=jsonpath='{.status.phase}'=Succeeded
kubectl -n tag04 logs regctl
kubectl -n tag04 delete pod regctl

Expected:

pod/regctl condition met
registry.tag04:5000/privat/whoami:1.0
1.0
pod "regctl" deleted from tag04 namespace

Break it:

# on cp1
kubectl -n tag04 create deployment intern --image=172.16.0.126:30500/privat/whoami:1.0

Symptom:

# on cp1
kubectl -n tag04 get pod -l app=intern \
  -o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.message}{"\n"}'

Expected (line wrapped, abridged):

Back-off pulling image "172.16.0.126:30500/privat/whoami:1.0": ErrImagePull: ...
  pull access denied, repository does not exist or may require authorization:
  authorization failed: no basic auth credentials

The pull reaches the registry, but the registry demands credentials. Without the entry for 172.16.0.126:30500 from Drill 3 it would not get this far, because containerd would talk to the registry over HTTPS. The message would then look like this:

Back-off pulling image "172.16.0.126:30500/privat/whoami:1.0": ErrImagePull: ...
  failed to do request: Head
  "https://172.16.0.126:30500/v2/privat/whoami/manifests/1.0": http: server gave
  HTTP response to HTTPS client

Fix:

# on cp1
kubectl -n tag04 create secret docker-registry intern-pull \
  --docker-server=172.16.0.126:30500 \
  --docker-username=labor --docker-password=labor-pw-123
kubectl -n tag04 patch deploy intern --type=merge \
  -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"intern-pull"}]}}}}'
kubectl -n tag04 rollout status deploy/intern

Verify:

# on cp1
kubectl -n tag04 events --for pod/$(kubectl -n tag04 get pod -l app=intern \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}') \
  | grep Pulled | cut -c1-88

Expected:

10s         Normal   Pulled      Pod/intern-7574d675f7-h684z   Successfully pulled image

Why? The Secret must live in the same namespace as the Pod, and --docker-server must match the registry part of the image name exactly – here including the port.

Save your progress? Sign up for free

Drill 5: CreateContainerConfigError

Break it:

# on cp1
cat > config.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata: {name: app-secret, namespace: tag04}
stringData: {db-password: lab-db-123}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: konfig, namespace: tag04}
spec:
  replicas: 1
  selector: {matchLabels: {app: konfig}}
  template:
    metadata: {labels: {app: konfig}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command: [sh, -c, 'echo "Modus $MODE"; sleep infinity']
        env:
        - name: MODE
          valueFrom: {configMapKeyRef: {name: app-config, key: mode}}
        - name: DB_PASSWORD
          valueFrom: {secretKeyRef: {name: app-secret, key: DB_PASSWORD}}
EOF
kubectl apply -f config.yaml

Symptom:

# on cp1
kubectl -n tag04 get pods -l app=konfig
kubectl -n tag04 events --for $(kubectl -n tag04 get pod -l app=konfig -o name) \
  --types=Warning | cut -c1-88

Expected (line wrapped):

NAME                      READY   STATUS                       RESTARTS   AGE
konfig-6dff46b849-5jvhk   0/1     CreateContainerConfigError   0          8s
LAST SEEN         TYPE      REASON   OBJECT                        MESSAGE
6s (x3 over 7s)   Warning   Failed   Pod/konfig-6dff46b849-5jvhk   Error: configmap
  "app-config" not found

Fix, part one:

# on cp1
kubectl -n tag04 create configmap app-config --from-literal=mode=produktion

Verify: The kubelet retries on its own – and finds the next error:

# on cp1
kubectl -n tag04 events --for $(kubectl -n tag04 get pod -l app=konfig -o name) \
  --types=Warning | tail -1 | cut -c1-88
kubectl -n tag04 get secret app-secret -o jsonpath='{.data}{"\n"}'

Expected (line wrapped):

9s                  Warning   Failed   Pod/konfig-6dff46b849-5jvhk   Error: couldn't
  find key DB_PASSWORD in Secret tag04/app-secret
{"db-password":"bGFiLWRiLTEyMw=="}

Cause: The Secret name is right, but the key is called db-password, not DB_PASSWORD. The kubelet only ever reports the first missing reference.

Fix, part two:

# on cp1
kubectl -n tag04 patch deploy konfig --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/containers/0/env/1/valueFrom/secretKeyRef/key",
  "value": "db-password"}]'
kubectl -n tag04 rollout status deploy/konfig
kubectl -n tag04 logs deploy/konfig

Expected (abridged):

deployment "konfig" successfully rolled out
Found 2 pods, using pod/konfig-866568954c-s6r94
Modus produktion

Save your progress? Sign up for free

Drill 6: exec format error – the missing shebang

Break it:

# on cp1
cat > skript.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: start-skript, namespace: tag04}
data:
  start.sh: |
    echo "Starte Dienst"
    exec sleep infinity
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: skript, namespace: tag04}
spec:
  replicas: 1
  selector: {matchLabels: {app: skript}}
  template:
    metadata: {labels: {app: skript}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command: [/scripts/start.sh]
        volumeMounts: [{name: skript, mountPath: /scripts}]
      volumes:
      - name: skript
        configMap: {name: start-skript, defaultMode: 0755}
EOF
kubectl apply -f skript.yaml

Symptom and investigation:

# on cp1
kubectl -n tag04 get pods -l app=skript
kubectl -n tag04 logs deploy/skript
kubectl -n tag04 get pod -l app=skript \
  -o jsonpath='{.items[0].status.containerStatuses[0].state.terminated.exitCode}{"\n"}'

Expected:

NAME                      READY   STATUS   RESTARTS      AGE
skript-7c8d47fd74-f62gt   0/1     Error    1 (10s ago)   12s
exec /scripts/start.sh: exec format error
255

Cause: The file is executable, but without #!/bin/sh the kernel does not know what to run it with. A shell would fall back to reading such a script itself; the kernel does not. Exit code 255 comes from the start process in the container.

Fix:

# on cp1
sed -i 's|  start.sh: \||  start.sh: \|\n    #!/bin/sh|' skript.yaml
sed -n 4,8p skript.yaml
kubectl apply -f skript.yaml
kubectl -n tag04 rollout restart deploy/skript
kubectl -n tag04 rollout status deploy/skript

Verify:

# on cp1
kubectl -n tag04 logs deploy/skript

Expected (abridged):

Starte Dienst

Why the rollout restart? The ConfigMap in the volume does get updated after a while, but a new Pod shows immediately whether the fix works.

Save your progress? Sign up for free

Drill 7: exec format error – foreign architecture

A multi-arch image is an index that points to a separate manifest per architecture. If you reference a manifest directly by its digest, you bypass that selection.

# on cp1
kubectl -n tag04 run digest --restart=Never --image=regclient/regctl:v0.11.7-alpine \
  --command -- sh -c 'for p in linux/amd64 linux/arm64; do
  echo "$p $(regctl image digest --platform $p busybox:1.37)"; done'
kubectl -n tag04 wait pod/digest --for=jsonpath='{.status.phase}'=Succeeded
kubectl -n tag04 logs digest
kubectl -n tag04 delete pod digest

Expected (abridged):

linux/amd64 sha256:66a6306db78bf2dbf3487f293aa8d6990d8e506fdffab9cc43fe422becf886e4
linux/arm64 sha256:d82c2ab94640ded77cf76514ce6a84870761105058a4a9e51b05a8a79be97a6c

Break it: On amd64 nodes you take the arm64 digest; on arm64 nodes (Apple Silicon, multipass) the amd64 digest.

# on cp1
D=sha256:d82c2ab94640ded77cf76514ce6a84870761105058a4a9e51b05a8a79be97a6c
kubectl -n tag04 create deployment fremd --image=busybox@$D -- sleep infinity

Symptom and investigation:

# on cp1
kubectl -n tag04 get pods -l app=fremd
kubectl -n tag04 logs deploy/fremd
kubectl -n tag04 get pod -l app=fremd -o jsonpath='{.items[0].spec.nodeName}{"\n"}'

Expected:

NAME                     READY   STATUS             RESTARTS      AGE
fremd-568d4f46d8-p9c7s   0/1     CrashLoopBackOff   1 (11s ago)   16s
exec /bin/sleep: exec format error
worker2
# on worker2
D=sha256:d82c2ab94640ded77cf76514ce6a84870761105058a4a9e51b05a8a79be97a6c
sudo -i crictl inspecti busybox@$D | jq -r .info.imageSpec.architecture

Expected:

arm64

Cause: containerd pulls whatever the digest asks for, even if it does not match the node. Only at start does the kernel fail on the foreign binary.

Fix:

# on cp1
kubectl -n tag04 set image deploy/fremd busybox=busybox:1.37
kubectl -n tag04 rollout status deploy/fremd

Verify:

# on cp1
kubectl -n tag04 get pods -l app=fremd

Expected: a new Pod 1/1 Running, the old one in Terminating.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show name and image for all Pods in tag04. Target time: 30 s
  2. Show all Pods in tag04 that are not Running. Target time: 20 s
  3. Which registry is the Secret intern-pull for? Target time: 45 s
  4. Show all busybox images on worker1. Target time: 20 s

Solution

# on cp1
kubectl -n tag04 get pods -o custom-columns=NAME:.metadata.name,\
IMAGE:.spec.containers[0].image
kubectl -n tag04 get pods --field-selector=status.phase!=Running
kubectl -n tag04 get secret intern-pull -o jsonpath='{.data.\.dockerconfigjson}' \
  | base64 -d | jq -r '.auths | keys[]'
ssh worker1 sudo -i crictl images | grep busybox

Expected (abridged):

NAME                          IMAGE
fremd-78996cf9-9bvcd          busybox:1.37
intern-7574d675f7-h684z       172.16.0.126:30500/privat/whoami:1.0
konfig-866568954c-s6r94       busybox:1.37
registry-6b5c56b54d-xbgmg     registry:3.1.2
skript-f5c7c985c-gqcs4        busybox:1.37
tippfehler-6f8bbbc746-8xzld   nginxinc/nginx-unprivileged:1.29-alpine
No resources found in tag04 namespace.
172.16.0.126:30500
docker.io/library/busybox                  1.37                   30ecbe1509090 ...

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag04 get deploy

Expected: fremd, intern, konfig, registry, skript and tippfehler, all 1/1.

# on worker2
sudo test -f /etc/rancher/rke2/registries.yaml && echo vorhanden

Expected:

vorhanden

Challenge

A Deployment offline uses nginxinc/nginx-unprivileged:1.29-alpine-slim with imagePullPolicy: Never, because the nodes will later run without internet access. Get it running without changing the manifest.

Hint: Never means the kubelet never pulls by itself; the image must already be on the node.

Solution

# on cp1
cat > offline.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: offline, namespace: tag04}
spec:
  replicas: 1
  selector: {matchLabels: {app: offline}}
  template:
    metadata: {labels: {app: offline}}
    spec:
      containers:
      - name: web
        image: nginxinc/nginx-unprivileged:1.29-alpine-slim
        imagePullPolicy: Never
EOF
kubectl apply -f offline.yaml
kubectl -n tag04 events --for $(kubectl -n tag04 get pod -l app=offline -o name) \
  --types=Warning | head -2 | cut -c1-88

Expected (line wrapped):

LAST SEEN         TYPE      REASON              OBJECT                         MESSAGE
5s (x3 over 7s)   Warning   ErrImageNeverPull   Pod/offline-748df9f596-bqx6b   Container
  image "nginxinc/nginx-unprivileged:1.29-alpine-slim" is not present with pull
  policy of Never

Put the image on all nodes, because the next Pod may land somewhere else:

# on cp1
sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpine-slim
ssh -n worker1 sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpine-slim
ssh -n worker2 sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpine-slim
kubectl -n tag04 get pods -l app=offline

Expected (after about 30 seconds):

NAME                       READY   STATUS    RESTARTS   AGE
offline-748df9f596-bqx6b   1/1     Running   0          66s

The kubelet retries on its own; there is no need to restart the Pod.

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag04
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

Expected:

namespace "tag04" deleted
No resources found

The registries.yaml stays on all nodes: from now on, the Docker Hub mirror spares your pull limit. The entry for 172.16.0.126:30500 does no harm without the registry.

Interview Note

  • ImagePullBackOff – how do you proceed? Read the message: not found (tag), no such host/timeout (the node’s network/DNS), pull access denied (private or does not exist at all), then crictl pull on the node.
  • Where do registry credentials go? As imagePullSecrets in the Pod or on the ServiceAccount, with the Secret in the same namespace; alternatively node-wide in registries.yaml.
  • exec format error? Missing shebang or wrong architecture – check with crictl inspecti or regctl image digest --platform.

Enterprise Note

  • Pull-through cache (Harbor, Zot, Artifactory) instead of a public mirror; in registries.yaml with credentials under configs:.
  • imagePullSecrets on the namespace’s default ServiceAccount save you from putting them in every manifest.
  • Pinning images by digest is good – but always the index digest, never that of a single architecture.
  • Registries without TLS only in the lab; in production use your own CA certificate under configs: tls: ca_file.
  • Air-gapped nodes get their images in advance (tarballs in agent/images/) and imagePullPolicy: IfNotPresent instead of Never.