Day 4: Images and Container Start
Goal: You tell pull errors apart by their message, check images directly on the node and find errors that happen between pull and start. · Duration: approx. 60 min · Prerequisite: Day 1
Briefing
- After scheduling, the kubelet takes over: pull the image (containerd), assemble the container from env, Secrets and volumes, start the process. Each phase has its own failure pattern.
ErrImagePullis the first failure,ImagePullBackOffis the wait until the next attempt. The cause is in the message behind it, not in the status.exec format erroris a start error, not a pull error: the image is there, but the kernel cannot execute the file.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Wrong tag | ErrImagePull |
Message: not found |
| Registry unreachable | ErrImagePull |
Message: no such host, HTTPS client |
| Private without Secret | ImagePullBackOff |
Message: pull access denied |
| ConfigMap/Secret missing | CreateContainerConfigError |
Event Failed on the Pod |
| Shebang, architecture | CrashLoopBackOff |
Log: exec format error |
Drill 1: Three pull errors, three messages
Break it:
# on cp1
mkdir -p ~/debug/tag04 && cd ~/debug/tag04
kubectl create namespace tag04
kubectl -n tag04 create deployment tippfehler \
--image=nginxinc/nginx-unprivileged:1.29-alpne
kubectl -n tag04 create deployment unerreichbar --image=registry.tag04.invalid/web:1.0
kubectl -n tag04 create deployment privat --image=docker.io/dbglabor/privat-app:1.0Symptom:
# on cp1
kubectl -n tag04 get podsExpected:
NAME READY STATUS RESTARTS AGE
privat-6b74c95474-p5shr 0/1 ImagePullBackOff 0 40s
tippfehler-7c975b5497-5cc46 0/1 ErrImagePull 0 40s
unerreichbar-699cd56b54-lmwrx 0/1 ErrImagePull 0 40s
Investigate: The message is in
state.waiting.message (or under Events in
describe):
# on cp1
for a in tippfehler unerreichbar privat; do
kubectl -n tag04 get pod -l app=$a \
-o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.message}{"\n"}'
doneExpected (lines wrapped, abridged):
rpc error: code = NotFound desc = failed to pull and unpack image
"docker.io/nginxinc/nginx-unprivileged:1.29-alpne": ...
docker.io/nginxinc/nginx-unprivileged:1.29-alpne: not found
failed to pull and unpack image "registry.tag04.invalid/web:1.0": ... failed to do
request: Head "https://registry.tag04.invalid/v2/web/manifests/1.0": dial tcp:
lookup registry.tag04.invalid on 127.0.0.53:53: no such host
Back-off pulling image "docker.io/dbglabor/privat-app:1.0": ErrImagePull: ... pull
access denied, repository does not exist or may require authorization: server
message: insufficient_scope: authorization failed
Cause: not found – the registry answers
but does not know the tag. no such host – the
node (not the Pod!) cannot resolve the name; the
address 127.0.0.53 is the host’s systemd-resolved.
pull access denied – the repository does not exist
or it is private; Docker Hub deliberately does not tell
the two apart.
Fix for the typo (with
create deployment, the container name comes from the image
name):
# on cp1
kubectl -n tag04 set image deploy/tippfehler \
nginx-unprivileged=nginxinc/nginx-unprivileged:1.29-alpine
kubectl -n tag04 rollout status deploy/tippfehler
kubectl -n tag04 delete deploy unerreichbar privatVerify:
# on cp1
kubectl -n tag04 get pods -l app=tippfehlerExpected: one Pod 1/1 Running. You
rebuild the private case properly in Drill 4.
Save your progress? Sign up for free
Drill 2: Check on the node
You check whether a pull works on the node itself with
crictl, without the kubelet. That separates “node cannot
reach the registry” from “Pod spec is wrong”.
# on worker1
sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpne 2>&1 | tail -1 | cut -c1-88
sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpine
sudo -i crictl images | grep -E 'IMAGE|nginx'Expected (abridged):
time="2026-11-20T07:09:08Z" level=error msg="pulling image: rpc error: code = NotFound d
Image is up to date for sha256:7312fa98595737958df41e20f21a2880dd9775f455c6bf76e638024dc
IMAGE TAG IMAGE ID ...
docker.io/nginxinc/nginx-unprivileged 1.29 b3c400d96d1d1 ...
docker.io/nginxinc/nginx-unprivileged 1.29-alpine 7312fa9859573 ...
crictl reports Image is up to date after
every successful pull. containerd expands short names like
nginxinc/… to docker.io/….
Talos: There the commands are
talosctl image pullandtalosctl image list(Day 18).
Save your progress? Sign up for free
Drill 3: registries.yaml – mirror and HTTP registry
Docker Hub limits anonymous pulls per IP. A mirror is not configured
in the Pod but in containerd on every node. RKE2 reads
/etc/rancher/rke2/registries.yaml for this – only at
startup. The second entry prepares Drill 4: the lab registry there only
speaks HTTP, but containerd tries HTTPS for every registry unless told
otherwise.
# on cp1, worker1 and worker2
sudo tee /etc/rancher/rke2/registries.yaml >/dev/null <<'EOF'
mirrors:
docker.io:
endpoint: ["https://mirror.gcr.io"]
"172.16.0.126:30500":
endpoint: ["http://172.16.0.126:30500"]
EOF
sudo chmod 600 /etc/rancher/rke2/registries.yamlWarning: Restarting the RKE2 service restarts the kubelet and containerd. Running Pods survive it, but on
cp1the API is gone for a moment.
# on cp1
sudo systemctl restart rke2-server# on worker1 and worker2
sudo systemctl restart rke2-agentFrom this, RKE2 generates a hosts.toml per registry for
containerd:
# on worker1
sudo cat /var/lib/rancher/rke2/agent/etc/containerd/certs.d/docker.io/hosts.tomlExpected:
# File generated by rke2. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]
[host]
[host."https://mirror.gcr.io/v2"]
capabilities = ["pull", "resolve"]
# on worker2
sudo -i crictl pull busybox:1.36Expected:
Image is up to date for sha256:b116e155074440ffd9e449559433feb4cd2341eb3554b1da1c638c
Why? containerd asks the mirror first and otherwise falls back to
server. A broken mirror therefore does not show up right
away: pulls just get slower and count against the limit again.
Save your progress? Sign up for free
Drill 4: Private registry and imagePullSecret
For the real path you need a registry with a password. You start
registry:3 with an htpasswd file in the cluster, reachable
on NodePort 30500.
# on cp1
kubectl -n tag04 run htpasswd --rm -i --quiet --restart=Never \
--image=httpd:2.4-alpine -- htpasswd -Bbn labor labor-pw-123 > htpasswd
kubectl -n tag04 create secret generic registry-auth --from-file=htpasswd
cat > registry.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: registry, namespace: tag04}
spec:
selector: {matchLabels: {app: registry}}
template:
metadata: {labels: {app: registry}}
spec:
containers:
- name: registry
image: registry:3.1.2
env:
- {name: REGISTRY_AUTH, value: htpasswd}
- {name: REGISTRY_AUTH_HTPASSWD_REALM, value: labor}
- {name: REGISTRY_AUTH_HTPASSWD_PATH, value: /auth/htpasswd}
ports: [{containerPort: 5000}]
volumeMounts: [{name: auth, mountPath: /auth}]
volumes:
- name: auth
secret: {secretName: registry-auth}
---
apiVersion: v1
kind: Service
metadata: {name: registry, namespace: tag04}
spec:
type: NodePort
selector: {app: registry}
ports: [{port: 5000, nodePort: 30500}]
EOF
kubectl apply -f registry.yaml
kubectl -n tag04 rollout status deploy/registry
curl -s -i http://172.16.0.126:30500/v2/ | grep -E 'HTTP|Authenticate'Expected:
HTTP/1.1 401 Unauthorized
Www-Authenticate: Basic realm="labor"
Copy an image into it, using regctl in a short-lived
Pod:
# on cp1
kubectl -n tag04 run regctl --restart=Never --image=regclient/regctl:v0.11.7-alpine \
--command -- sh -c '
regctl registry set --tls disabled registry.tag04:5000 2>/dev/null
regctl registry login registry.tag04:5000 -u labor -p labor-pw-123 &&
regctl image copy traefik/whoami:v1.11 registry.tag04:5000/privat/whoami:1.0 &&
regctl tag ls registry.tag04:5000/privat/whoami'
kubectl -n tag04 wait pod/regctl --for=jsonpath='{.status.phase}'=Succeeded
kubectl -n tag04 logs regctl
kubectl -n tag04 delete pod regctlExpected:
pod/regctl condition met
registry.tag04:5000/privat/whoami:1.0
1.0
pod "regctl" deleted from tag04 namespace
Break it:
# on cp1
kubectl -n tag04 create deployment intern --image=172.16.0.126:30500/privat/whoami:1.0Symptom:
# on cp1
kubectl -n tag04 get pod -l app=intern \
-o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.message}{"\n"}'Expected (line wrapped, abridged):
Back-off pulling image "172.16.0.126:30500/privat/whoami:1.0": ErrImagePull: ...
pull access denied, repository does not exist or may require authorization:
authorization failed: no basic auth credentials
The pull reaches the registry, but the registry demands credentials.
Without the entry for 172.16.0.126:30500 from Drill 3 it
would not get this far, because containerd would talk to the registry
over HTTPS. The message would then look like this:
Back-off pulling image "172.16.0.126:30500/privat/whoami:1.0": ErrImagePull: ...
failed to do request: Head
"https://172.16.0.126:30500/v2/privat/whoami/manifests/1.0": http: server gave
HTTP response to HTTPS client
Fix:
# on cp1
kubectl -n tag04 create secret docker-registry intern-pull \
--docker-server=172.16.0.126:30500 \
--docker-username=labor --docker-password=labor-pw-123
kubectl -n tag04 patch deploy intern --type=merge \
-p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"intern-pull"}]}}}}'
kubectl -n tag04 rollout status deploy/internVerify:
# on cp1
kubectl -n tag04 events --for pod/$(kubectl -n tag04 get pod -l app=intern \
--field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}') \
| grep Pulled | cut -c1-88Expected:
10s Normal Pulled Pod/intern-7574d675f7-h684z Successfully pulled image
Why? The Secret must live in the same namespace as
the Pod, and --docker-server must match the registry part
of the image name exactly – here including the port.
Save your progress? Sign up for free
Drill 5: CreateContainerConfigError
Break it:
# on cp1
cat > config.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata: {name: app-secret, namespace: tag04}
stringData: {db-password: lab-db-123}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: konfig, namespace: tag04}
spec:
replicas: 1
selector: {matchLabels: {app: konfig}}
template:
metadata: {labels: {app: konfig}}
spec:
containers:
- name: app
image: busybox:1.37
command: [sh, -c, 'echo "Modus $MODE"; sleep infinity']
env:
- name: MODE
valueFrom: {configMapKeyRef: {name: app-config, key: mode}}
- name: DB_PASSWORD
valueFrom: {secretKeyRef: {name: app-secret, key: DB_PASSWORD}}
EOF
kubectl apply -f config.yamlSymptom:
# on cp1
kubectl -n tag04 get pods -l app=konfig
kubectl -n tag04 events --for $(kubectl -n tag04 get pod -l app=konfig -o name) \
--types=Warning | cut -c1-88Expected (line wrapped):
NAME READY STATUS RESTARTS AGE
konfig-6dff46b849-5jvhk 0/1 CreateContainerConfigError 0 8s
LAST SEEN TYPE REASON OBJECT MESSAGE
6s (x3 over 7s) Warning Failed Pod/konfig-6dff46b849-5jvhk Error: configmap
"app-config" not found
Fix, part one:
# on cp1
kubectl -n tag04 create configmap app-config --from-literal=mode=produktionVerify: The kubelet retries on its own – and finds the next error:
# on cp1
kubectl -n tag04 events --for $(kubectl -n tag04 get pod -l app=konfig -o name) \
--types=Warning | tail -1 | cut -c1-88
kubectl -n tag04 get secret app-secret -o jsonpath='{.data}{"\n"}'Expected (line wrapped):
9s Warning Failed Pod/konfig-6dff46b849-5jvhk Error: couldn't
find key DB_PASSWORD in Secret tag04/app-secret
{"db-password":"bGFiLWRiLTEyMw=="}
Cause: The Secret name is right, but the key is
called db-password, not DB_PASSWORD. The
kubelet only ever reports the first missing reference.
Fix, part two:
# on cp1
kubectl -n tag04 patch deploy konfig --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/containers/0/env/1/valueFrom/secretKeyRef/key",
"value": "db-password"}]'
kubectl -n tag04 rollout status deploy/konfig
kubectl -n tag04 logs deploy/konfigExpected (abridged):
deployment "konfig" successfully rolled out
Found 2 pods, using pod/konfig-866568954c-s6r94
Modus produktion
Save your progress? Sign up for free
Drill 6: exec format error – the missing shebang
Break it:
# on cp1
cat > skript.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: start-skript, namespace: tag04}
data:
start.sh: |
echo "Starte Dienst"
exec sleep infinity
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: skript, namespace: tag04}
spec:
replicas: 1
selector: {matchLabels: {app: skript}}
template:
metadata: {labels: {app: skript}}
spec:
containers:
- name: app
image: busybox:1.37
command: [/scripts/start.sh]
volumeMounts: [{name: skript, mountPath: /scripts}]
volumes:
- name: skript
configMap: {name: start-skript, defaultMode: 0755}
EOF
kubectl apply -f skript.yamlSymptom and investigation:
# on cp1
kubectl -n tag04 get pods -l app=skript
kubectl -n tag04 logs deploy/skript
kubectl -n tag04 get pod -l app=skript \
-o jsonpath='{.items[0].status.containerStatuses[0].state.terminated.exitCode}{"\n"}'Expected:
NAME READY STATUS RESTARTS AGE
skript-7c8d47fd74-f62gt 0/1 Error 1 (10s ago) 12s
exec /scripts/start.sh: exec format error
255
Cause: The file is executable, but without
#!/bin/sh the kernel does not know what to run it with. A
shell would fall back to reading such a script itself; the kernel does
not. Exit code 255 comes from the start process in the container.
Fix:
# on cp1
sed -i 's| start.sh: \|| start.sh: \|\n #!/bin/sh|' skript.yaml
sed -n 4,8p skript.yaml
kubectl apply -f skript.yaml
kubectl -n tag04 rollout restart deploy/skript
kubectl -n tag04 rollout status deploy/skriptVerify:
# on cp1
kubectl -n tag04 logs deploy/skriptExpected (abridged):
Starte Dienst
Why the rollout restart? The ConfigMap in the volume
does get updated after a while, but a new Pod shows immediately whether
the fix works.
Save your progress? Sign up for free
Drill 7: exec format error – foreign architecture
A multi-arch image is an index that points to a separate manifest per architecture. If you reference a manifest directly by its digest, you bypass that selection.
# on cp1
kubectl -n tag04 run digest --restart=Never --image=regclient/regctl:v0.11.7-alpine \
--command -- sh -c 'for p in linux/amd64 linux/arm64; do
echo "$p $(regctl image digest --platform $p busybox:1.37)"; done'
kubectl -n tag04 wait pod/digest --for=jsonpath='{.status.phase}'=Succeeded
kubectl -n tag04 logs digest
kubectl -n tag04 delete pod digestExpected (abridged):
linux/amd64 sha256:66a6306db78bf2dbf3487f293aa8d6990d8e506fdffab9cc43fe422becf886e4
linux/arm64 sha256:d82c2ab94640ded77cf76514ce6a84870761105058a4a9e51b05a8a79be97a6c
Break it: On amd64 nodes you take the arm64 digest; on arm64 nodes (Apple Silicon, multipass) the amd64 digest.
# on cp1
D=sha256:d82c2ab94640ded77cf76514ce6a84870761105058a4a9e51b05a8a79be97a6c
kubectl -n tag04 create deployment fremd --image=busybox@$D -- sleep infinitySymptom and investigation:
# on cp1
kubectl -n tag04 get pods -l app=fremd
kubectl -n tag04 logs deploy/fremd
kubectl -n tag04 get pod -l app=fremd -o jsonpath='{.items[0].spec.nodeName}{"\n"}'Expected:
NAME READY STATUS RESTARTS AGE
fremd-568d4f46d8-p9c7s 0/1 CrashLoopBackOff 1 (11s ago) 16s
exec /bin/sleep: exec format error
worker2
# on worker2
D=sha256:d82c2ab94640ded77cf76514ce6a84870761105058a4a9e51b05a8a79be97a6c
sudo -i crictl inspecti busybox@$D | jq -r .info.imageSpec.architectureExpected:
arm64
Cause: containerd pulls whatever the digest asks for, even if it does not match the node. Only at start does the kernel fail on the foreign binary.
Fix:
# on cp1
kubectl -n tag04 set image deploy/fremd busybox=busybox:1.37
kubectl -n tag04 rollout status deploy/fremdVerify:
# on cp1
kubectl -n tag04 get pods -l app=fremdExpected: a new Pod 1/1 Running, the
old one in Terminating.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show name and image for all Pods in
tag04. Target time: 30 s - Show all Pods in
tag04that are notRunning. Target time: 20 s - Which registry is the Secret
intern-pullfor? Target time: 45 s - Show all busybox images on
worker1. Target time: 20 s
Solution
# on cp1
kubectl -n tag04 get pods -o custom-columns=NAME:.metadata.name,\
IMAGE:.spec.containers[0].image
kubectl -n tag04 get pods --field-selector=status.phase!=Running
kubectl -n tag04 get secret intern-pull -o jsonpath='{.data.\.dockerconfigjson}' \
| base64 -d | jq -r '.auths | keys[]'
ssh worker1 sudo -i crictl images | grep busyboxExpected (abridged):
NAME IMAGE
fremd-78996cf9-9bvcd busybox:1.37
intern-7574d675f7-h684z 172.16.0.126:30500/privat/whoami:1.0
konfig-866568954c-s6r94 busybox:1.37
registry-6b5c56b54d-xbgmg registry:3.1.2
skript-f5c7c985c-gqcs4 busybox:1.37
tippfehler-6f8bbbc746-8xzld nginxinc/nginx-unprivileged:1.29-alpine
No resources found in tag04 namespace.
172.16.0.126:30500
docker.io/library/busybox 1.37 30ecbe1509090 ...
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag04 get deployExpected: fremd, intern,
konfig, registry, skript and
tippfehler, all 1/1.
# on worker2
sudo test -f /etc/rancher/rke2/registries.yaml && echo vorhandenExpected:
vorhanden
Challenge
A Deployment offline uses
nginxinc/nginx-unprivileged:1.29-alpine-slim with
imagePullPolicy: Never, because the nodes will later run
without internet access. Get it running without changing the
manifest.
Hint: Never means the kubelet never
pulls by itself; the image must already be on the node.
Solution
# on cp1
cat > offline.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: offline, namespace: tag04}
spec:
replicas: 1
selector: {matchLabels: {app: offline}}
template:
metadata: {labels: {app: offline}}
spec:
containers:
- name: web
image: nginxinc/nginx-unprivileged:1.29-alpine-slim
imagePullPolicy: Never
EOF
kubectl apply -f offline.yaml
kubectl -n tag04 events --for $(kubectl -n tag04 get pod -l app=offline -o name) \
--types=Warning | head -2 | cut -c1-88Expected (line wrapped):
LAST SEEN TYPE REASON OBJECT MESSAGE
5s (x3 over 7s) Warning ErrImageNeverPull Pod/offline-748df9f596-bqx6b Container
image "nginxinc/nginx-unprivileged:1.29-alpine-slim" is not present with pull
policy of Never
Put the image on all nodes, because the next Pod may land somewhere else:
# on cp1
sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpine-slim
ssh -n worker1 sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpine-slim
ssh -n worker2 sudo -i crictl pull nginxinc/nginx-unprivileged:1.29-alpine-slim
kubectl -n tag04 get pods -l app=offlineExpected (after about 30 seconds):
NAME READY STATUS RESTARTS AGE
offline-748df9f596-bqx6b 1/1 Running 0 66s
The kubelet retries on its own; there is no need to restart the Pod.
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag04
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=SucceededExpected:
namespace "tag04" deleted
No resources found
The registries.yaml stays on all nodes: from now on, the
Docker Hub mirror spares your pull limit. The entry for
172.16.0.126:30500 does no harm without the registry.
Interview Note
ImagePullBackOff– how do you proceed? Read the message:not found(tag),no such host/timeout (the node’s network/DNS),pull access denied(private or does not exist at all), thencrictl pullon the node.- Where do registry credentials go? As
imagePullSecretsin the Pod or on the ServiceAccount, with the Secret in the same namespace; alternatively node-wide inregistries.yaml. exec format error? Missing shebang or wrong architecture – check withcrictl inspectiorregctl image digest --platform.
Enterprise Note
- Pull-through cache (Harbor, Zot, Artifactory) instead of a public
mirror; in
registries.yamlwith credentials underconfigs:. imagePullSecretson the namespace’sdefaultServiceAccount save you from putting them in every manifest.- Pinning images by digest is good – but always the index digest, never that of a single architecture.
- Registries without TLS only in the lab; in production use your own
CA certificate under
configs: tls: ca_file. - Air-gapped nodes get their images in advance (tarballs in
agent/images/) andimagePullPolicy: IfNotPresentinstead ofNever.