Skip to content
Hands-on IT IT Drills

Day 18: Competition – Broken Deployments

Goal: You find and fix ten typical errors in eight broken Deployments – against the clock, with nothing but get, describe, logs and events. · Duration: approx. 75 min · Prerequisite: Days 0–17

Briefing

All five curriculum domains, with a focus on CKAD: Application Observability and Maintenance (15 %) – “Debugging in Kubernetes”. Eight teams have rolled out notes-api, and each one reports “it doesn’t work”. Each round gets its own namespace tag18-r1 … tag18-r8.

Status/symptom                 first look
ImagePullBackOff               k get events --field-selector type=Warning
Error / CrashLoopBackOff       k logs (without --previous!)
CreateContainerConfigError     k describe pod | tail
0/1 Running                    Events "Unhealthy"
Service not answering          k get endpointslices
OOMKilled                      k describe pod | grep -A3 "Last State"
Deployment 0/N, no Pods        Events on the ReplicaSet "FailedCreate"
Ingress 404 / 504              k describe ingress, NetworkPolicies
mkdir -p ~/ckad/tag18 && cd ~/ckad/tag18

Rules of Play

  • Per round: create the namespace, apply the manifest, then start the clock. Do not study the manifest – in the exam, too, you only read it once the status leads you there.
  • Find the cause, fix it, prove it with one command. Only then stop the clock.
  • Only read the Hint once the target time has run out.
  • Points are listed in the scoring at the end; write down your times.

Every round starts the same way (here for round 1):

k create ns tag18-r1
k config set-context --current --namespace=tag18-r1

Round 1: The image doesn’t arrive

Situation: Team 1 has rolled out notes-api 1.0. k get deploy notes has been showing 0/2 for minutes.

k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:v1.0
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
EOF

Target time: 2 min

Hint: Which tags does the registry really know?

Solution

k get pods
k get events --field-selector type=Warning | head -3
curl -s 172.16.0.126:30500/v2/notes-api/tags/list; echo

Expected (abridged):

NAME                     READY   STATUS             RESTARTS   AGE
notes-7c9c67bdc7-b74cv   0/1     ImagePullBackOff   0          26s
notes-7c9c67bdc7-h9544   0/1     ImagePullBackOff   0          26s
LAST SEEN   TYPE      REASON   OBJECT                       MESSAGE
10s         Warning   Failed   pod/notes-7c9c67bdc7-b74cv   Failed to pull image
"172.16.0.126:30500/notes-api:v1.0": rpc error: code = NotFound ... not found
10s         Warning   Failed   pod/notes-7c9c67bdc7-b74cv   Error: ErrImagePull
{"name":"notes-api","tags":["1.0","1.1"]}

The tag is 1.0, not v1.0:

k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.0
k rollout status deploy/notes

Expected (abridged):

deployment.apps/notes image updated
...
deployment "notes" successfully rolled out

code = NotFound means: registry reached, tag missing. With a wrong registry address, you would see connection refused or no such host there.

Save your progress? Sign up for free

Round 2: The key has a different name

Situation: Team 2 neatly puts the API key into a Secret. The Pod still keeps restarting.

k create ns tag18-r2
k config set-context --current --namespace=tag18-r2
k apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: notes-key
stringData:
  API_KEY: lab-key-123
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        envFrom:
        - secretRef:
            name: notes-key
EOF

Target time: 3 min

Hint: envFrom takes the Secret’s keys as variable names – exactly as they are written there.

Solution

k get pods
k logs deploy/notes
k logs deploy/notes --previous
k get secret notes-key -o jsonpath='{.data}'; echo

Expected (abridged):

NAME                    READY   STATUS   RESTARTS      AGE
notes-b675577f9-5qbdk   0/1     Error    2 (37s ago)   41s
time=2026-11-18T16:13:48.932Z level=ERROR msg="start abgebrochen" err="NOTES_API_KEY
oder NOTES_API_KEY_FILE fehlt"
unable to retrieve container logs for containerd://222bba0c36db…
{"API_KEY":"bGFiLWtleS0xMjM="}

Warning: In the lab (Kubernetes 1.37.2, containerd 2.4.2), --previous only returns unable to retrieve container logs for a Pod stuck in CrashLoopBackOff. k logs without --previous shows the last finished run – that is enough.

The Secret provides API_KEY, the app wants NOTES_API_KEY. Instead of changing the Secret (other teams might use it), --prefix maps the name:

k set env deploy/notes --from=secret/notes-key --prefix=NOTES_
k rollout status deploy/notes
k set env deploy/notes --list

Expected (abridged):

deployment.apps/notes env updated
deployment "notes" successfully rolled out
# Deployment notes, container notes-api
# NOTES_API_KEY from secret notes-key, key API_KEY

k set env --from writes one secretKeyRef entry per key into the Deployment.

Save your progress? Sign up for free

Round 3: The ConfigMap is missing

Situation: Team 3 has split port and key: the port via the ConfigMap notes-config, the key directly. The Pod never gets past startup.

k create ns tag18-r3
k config set-context --current --namespace=tag18-r3
k apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: notes-config
data:
  PORT: "8080"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
        envFrom:
        - configMapRef:
            name: notes-conf
EOF

Target time: 2 min

Hint: Compare the name in the error with k get cm.

Solution

k get pods
k describe pod -l app=notes | tail -1
k get cm

Expected (abridged):

NAME                     READY   STATUS                       RESTARTS   AGE
notes-6b679fcc6b-nzfkl   0/1     CreateContainerConfigError   0          20s
  Warning  Failed  6s (x4 over 19s)  kubelet  spec.containers{notes-api}: Error:
configmap "notes-conf" not found
NAME               DATA   AGE
kube-root-ca.crt   1      22s
notes-config       1      21s

In the exam, k edit deploy notes and notes-conf → notes-config is enough. Without an editor, use a JSON patch (lists without a merge key, such as envFrom, can only be changed precisely this way):

k patch deploy notes --type=json -p='[{"op": "replace",
  "path": "/spec/template/spec/containers/0/envFrom/0/configMapRef/name",
  "value": "notes-config"}]'
k rollout status deploy/notes

Expected (abridged):

deployment.apps/notes patched
deployment "notes" successfully rolled out

A missing key in an existing ConfigMap (configMapKeyRef) also ends in CreateContainerConfigError, just with couldn't find key.

Save your progress? Sign up for free

Round 4: The Service is silent

Situation: Team 4 has a Deployment and a Service. The test Pod client gets no answer from notes. Watch out: there are two errors.

k create ns tag18-r4
k config set-context --current --namespace=tag18-r4
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8081
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  selector:
    app: notes-api
  ports:
  - port: 80
    targetPort: 8080
EOF
k run client --image=busybox:1.37 -- sleep 3600

Target time: 4 min

Hint: An endpoint needs two things: a matching label and ready=true.

Solution

k get pods
k exec client -- wget -qO- -T3 notes/healthz
k get endpointslices -l kubernetes.io/service-name=notes
k get svc notes -o wide
k get pods -l app=notes --show-labels

Expected (abridged):

NAME                     READY   STATUS    RESTARTS   AGE
client                   1/1     Running   0          30s
notes-548bf6d74d-256fw   0/1     Running   0          31s
notes-548bf6d74d-5p4p8   0/1     Running   0          31s
wget: can't connect to remote host (10.107.39.198): Connection refused
command terminated with exit code 1
NAME          ADDRESSTYPE   PORTS     ENDPOINTS   AGE
notes-46zvl   IPv4          <unset>   <unset>     33s
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
notes   ClusterIP   10.107.39.198   <none>        80/TCP    33s   app=notes-api
NAME                     READY   STATUS    RESTARTS   AGE   LABELS
notes-548bf6d74d-256fw   0/1     Running   0          33s   app=notes,pod-template-...

Error 1: the selector looks for app=notes-api, the Pods are labeled app=notes.

k set selector svc notes app=notes
k get endpointslices -l kubernetes.io/service-name=notes \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]} '\
'{.conditions.ready}{"\n"}{end}'
k exec client -- wget -qO- -T3 notes/healthz

Expected:

service/notes selector updated
192.168.235.190 false
192.168.189.79 false
wget: can't connect to remote host (10.107.39.198): Connection refused
command terminated with exit code 1

Error 2: the Pods are now in the slice, but not ready – 0/1 Running was the second clue.

k get events --field-selector reason=Unhealthy | tail -1
k patch deploy notes --type=json -p='[{"op": "replace",
  "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port",
  "value": 8080}]'
k rollout status deploy/notes
k exec client -- wget -qO- -T3 notes/healthz; echo

Expected (abridged):

4s   Warning   Unhealthy   pod/notes-548bf6d74d-5p4p8   Readiness probe failed: Get
"http://192.168.189.79:8081/healthz": dial tcp 192.168.189.79:8081: connect: connection
refused
deployment.apps/notes patched
deployment "notes" successfully rolled out
ok

With Calico, Connection refused at the ClusterIP means: a Service without ready endpoints. A timeout points more toward a NetworkPolicy (round 8).

Save your progress? Sign up for free

Round 5: The export dies silently

Situation: Team 5 runs an export job as a Deployment. It keeps restarting, and k logs stays empty.

k create ns tag18-r5
k config set-context --current --namespace=tag18-r5
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: export
spec:
  replicas: 1
  selector:
    matchLabels:
      app: export
  template:
    metadata:
      labels:
        app: export
    spec:
      containers:
      - name: export
        image: busybox:1.37
        command: ["sh", "-c", "head -c 80m /dev/zero | tail > /dev/null;
          echo export fertig; sleep 3600"]
        resources:
          requests:
            memory: 32Mi
          limits:
            memory: 32Mi
EOF

Target time: 2 min

Hint: Empty logs plus restarts – the reason is in the status of the last run.

Solution

k get pods
k logs deploy/export
k describe pod -l app=export | grep -A3 "Last State"

Expected (abridged):

NAME                     READY   STATUS      RESTARTS      AGE
export-b689db759-p7c8t   0/1     OOMKilled   2 (25s ago)   30s
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

tail keeps the 80 MiB without a line ending in memory, while the limit allows 32 MiB. The kernel kills the container before it writes a single line.

k set resources deploy export --requests=memory=128Mi --limits=memory=128Mi
k rollout status deploy/export
k logs deploy/export

Expected (abridged):

deployment.apps/export resource requirements updated
deployment "export" successfully rolled out
export fertig

Save your progress? Sign up for free

Round 6: No Pods, no error on the Deployment

Situation: Team 6 does not see a single Pod in its new namespace. k get pods says No resources found.

k create ns tag18-r6
k config set-context --current --namespace=tag18-r6
k apply -f - <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.memory: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 3
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
EOF

Target time: 3 min

Hint: Pods are created by the ReplicaSet – that is where the events are.

Solution

k get deploy,rs
k get events --field-selector reason=FailedCreate | tail -1
k describe quota team

Expected (abridged):

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/notes   0/3     0            0           10s
NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/notes-5fdf9f698c   3         0         0       10s
1s   Warning   FailedCreate   replicaset/notes-5fdf9f698c   (combined from similar
events): Error creating: pods "notes-5fdf9f698c-4s7hm" is forbidden: failed quota: team:
must specify limits.memory for: notes-api; requests.cpu for: notes-api; requests.memory
for: notes-api
Resource         Used  Hard
--------         ----  ----
limits.memory    0     1Gi
requests.cpu     0     1
requests.memory  0     1Gi
k set resources deploy notes --requests=cpu=100m,memory=64Mi --limits=memory=64Mi
k rollout status deploy/notes
k describe quota team | tail -3

Expected (abridged):

deployment.apps/notes resource requirements updated
deployment "notes" successfully rolled out
limits.memory    192Mi  1Gi
requests.cpu     300m   1
requests.memory  192Mi  1Gi

DESIRED 3, CURRENT 0 is the telltale sign: the controller wants Pods, the API server refuses. A LimitRange with default values (Day 7) would have prevented the error.

Save your progress? Sign up for free

Round 7: The sidecar isn’t allowed

Situation: Team 7 has hardened notes-api with runAsNonRoot and added a small health poller. The Pod stays at 1/2.

k create ns tag18-r7
k config set-context --current --namespace=tag18-r7
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      securityContext:
        runAsNonRoot: true
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
      - name: poller
        image: busybox:1.37
        command: ["sh", "-c", "while true; do
          wget -qO- localhost:8080/healthz; echo; sleep 10; done"]
EOF

Target time: 3 min

Hint: runAsNonRoot forbids root, but does not set a user by itself.

Solution

k get pods
k get events --field-selector type=Warning | tail -1

Expected (abridged):

NAME                     READY   STATUS                       RESTARTS   AGE
notes-546897878f-fqkzn   1/2     CreateContainerConfigError   0          20s
5s   Warning   Failed   pod/notes-546897878f-fqkzn   Error: container has runAsNonRoot
and image will run as root (pod: "notes-546897878f-fqkzn_tag18-r7(9331…)", container:
poller)

notes-api runs as 65532 and passes the check, busybox runs as root. The hardening stays; the poller gets its own user:

k patch deploy notes --type=json -p='[{"op": "add",
  "path": "/spec/template/spec/containers/1/securityContext",
  "value": {"runAsUser": 65532}}]'
k rollout status deploy/notes
k logs deploy/notes -c poller

Expected (abridged):

deployment.apps/notes patched
deployment "notes" successfully rolled out
ok

Removing runAsNonRoot would be wrong – in the exam, you would lose the points for the security requirement.

Save your progress? Sign up for free

Round 8: Nothing gets in from outside

Situation: Team 8 publishes notes-api through Traefik as notes.tag18.example. The namespace has a default-deny policy. Nothing gets in from outside – again two errors.

k create ns tag18-r8
k config set-context --current --namespace=tag18-r8
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  selector:
    app: notes
  ports:
  - name: http
    port: 80
    targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: notes
spec:
  ingressClassName: traefik
  rules:
  - host: notes.tag18.example
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: notes
            port:
              number: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
EOF

Target time: 5 min

Hint: The Ingress points to the Service port, not to the container port. After that: from which namespace does Traefik’s traffic come?

Solution

curl -s -H 'Host: notes.tag18.example' http://172.16.0.126:30080/healthz
k describe ingress notes | grep -A3 Rules
k get svc notes
k logs -n traefik deploy/traefik --since=60s | grep tag18 | tail -1

Expected (abridged, color codes removed):

404 page not found
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  notes.tag18.example
                       /   notes:8080 ()
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
notes   ClusterIP   10.108.142.39   <none>        80/TCP    20s
… ERR Cannot create service error="service port not found" ingress=notes
namespace=tag18-r8 … serviceName=notes
servicePort=&ServiceBackendPort{Name:,Number:8080,}

Empty parentheses after the backend: the Service has no port 8080. Fix error 1:

k patch ingress notes --type=json -p='[{"op": "replace",
  "path": "/spec/rules/0/http/paths/0/backend/service/port/number", "value": 80}]'
k describe ingress notes | grep ' /'
curl -s -w '\n%{http_code}\n' -H 'Host: notes.tag18.example' \
  http://172.16.0.126:30080/healthz

Expected (the answer takes 30 s):

ingress.networking.k8s.io/notes patched
                       /   notes:80 (192.168.189.76:8080,192.168.235.165:8080)
Gateway Timeout
504

Error 2: Traefik cannot reach the Pods – default-deny blocks all incoming traffic. You specifically allow the namespace traefik on port 8080 (Pod port, not Service port):

k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: notes-from-traefik
spec:
  podSelector:
    matchLabels:
      app: notes
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: traefik
    ports:
    - port: 8080
EOF
curl -s -H 'Host: notes.tag18.example' -H 'X-API-Key: lab-key-123' \
  -d '{"text":"gewonnen"}' http://172.16.0.126:30080/notes; echo

Expected:

networkpolicy.networking.k8s.io/notes-from-traefik created
{"id":"4d28a99e-ba4b-4613-8da2-43d02bd1501e","text":"gewonnen"}

404 from Traefik means: no matching route. 504 means: the route is there, but the backend does not answer – typical for a NetworkPolicy.

Stop the clock.

Save your progress? Sign up for free

Scoring

Round Failure pattern Target time Points
1 ImagePullBackOff (tag) 2 min 1
2 CrashLoop, variable has a different name 3 min 2
3 CreateContainerConfigError (ConfigMap) 2 min 1
4 Selector + readiness port 4 min 2
5 OOMKilled 2 min 1
6 Quota, FailedCreate 3 min 2
7 runAsNonRoot vs. root image 3 min 2
8 Ingress port + NetworkPolicy 5 min 3

You only get full points within the target time and without a hint; with a hint or over time, half (round down). Overall target time: 24 min, maximum: 14 points.

Points Assessment
12–14 exam-ready – move on to the Practice Exam (Appendix B)
8–11 good; repeat the days of the rounds you missed
0–7 work through Days 14 and 15 once more

Checkpoint

k get deploy -A | grep tag18

Expected: eight lines from tag18-r1 to tag18-r8, each with all replicas ready (2/2, 1/1, 1/1, 2/2, 1/1, 3/3, 1/1, 2/2).

k get pods -A --no-headers | grep tag18 | grep -vc Running

Expected: 0

k exec -n tag18-r4 client -- wget -qO- -T3 notes/healthz; echo

Expected: ok

curl -s -H 'Host: notes.tag18.example' http://172.16.0.126:30080/version; echo

Expected: 1.0

Challenge

Build your own round for a study partner with two errors from two different days, write down the situation in two sentences and let them play. Good candidates: a liveness probe on the wrong path (Day 14), a secretKeyRef with the wrong key (Day 6), a rollout with maxUnavailable: 0 and a broken image (Day 11).

Hint: Test the round yourself first and note down the proof command.

Solution

There is no fixed solution. Afterwards, check together with the checkpoint commands whether everything is running again.

Clean up:

k config set-context --current --namespace=default
k delete ns tag18-r1 tag18-r2 tag18-r3 tag18-r4 tag18-r5 tag18-r6 tag18-r7 tag18-r8

Save your progress? Sign up for free

Exam Tip

  • Always proceed the same way: k get → events/describe → logs. The status column almost always tells you which of the three tools has the answer.
  • Finish every task with a proof command – that is the only way you notice the second error (rounds 4 and 8).
  • Fix the error in the existing object, don’t reinvent the object: names, labels and security requirements from the task stay as they are.

Enterprise Note

  • Most of today’s errors are caught by a CI check before deploying: kubeconform for the schema, kube-linter or policies (Kyverno, OPA) for probes, resources and securityContext.
  • Alerts on kube_deployment_status_replicas_unavailable and kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} report broken Deployments before a user does.
  • Pin image tags in production by digest; a typo in the tag then already shows up when the pipeline resolves it.