Day 18: Competition – Broken Deployments
Goal: You find and fix ten typical errors in eight broken Deployments – against the clock, with nothing but
get,describe,logsand events. · Duration: approx. 75 min · Prerequisite: Days 0–17
Briefing
All five curriculum domains, with a focus on CKAD: Application
Observability and Maintenance (15 %) – “Debugging in Kubernetes”. Eight
teams have rolled out notes-api, and each one reports “it doesn’t work”.
Each round gets its own namespace tag18-r1 …
tag18-r8.
Status/symptom first look
ImagePullBackOff k get events --field-selector type=Warning
Error / CrashLoopBackOff k logs (without --previous!)
CreateContainerConfigError k describe pod | tail
0/1 Running Events "Unhealthy"
Service not answering k get endpointslices
OOMKilled k describe pod | grep -A3 "Last State"
Deployment 0/N, no Pods Events on the ReplicaSet "FailedCreate"
Ingress 404 / 504 k describe ingress, NetworkPolicies
mkdir -p ~/ckad/tag18 && cd ~/ckad/tag18Rules of Play
- Per round: create the namespace, apply the manifest, then start the clock. Do not study the manifest – in the exam, too, you only read it once the status leads you there.
- Find the cause, fix it, prove it with one command. Only then stop the clock.
- Only read the Hint once the target time has run out.
- Points are listed in the scoring at the end; write down your times.
Every round starts the same way (here for round 1):
k create ns tag18-r1
k config set-context --current --namespace=tag18-r1Round 1: The image doesn’t arrive
Situation: Team 1 has rolled out notes-api 1.0.
k get deploy notes has been showing 0/2 for
minutes.
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 2
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:v1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
EOFTarget time: 2 min
Hint: Which tags does the registry really know?
Solution
k get pods
k get events --field-selector type=Warning | head -3
curl -s 172.16.0.126:30500/v2/notes-api/tags/list; echoExpected (abridged):
NAME READY STATUS RESTARTS AGE
notes-7c9c67bdc7-b74cv 0/1 ImagePullBackOff 0 26s
notes-7c9c67bdc7-h9544 0/1 ImagePullBackOff 0 26s
LAST SEEN TYPE REASON OBJECT MESSAGE
10s Warning Failed pod/notes-7c9c67bdc7-b74cv Failed to pull image
"172.16.0.126:30500/notes-api:v1.0": rpc error: code = NotFound ... not found
10s Warning Failed pod/notes-7c9c67bdc7-b74cv Error: ErrImagePull
{"name":"notes-api","tags":["1.0","1.1"]}
The tag is 1.0, not v1.0:
k set image deploy/notes notes-api=172.16.0.126:30500/notes-api:1.0
k rollout status deploy/notesExpected (abridged):
deployment.apps/notes image updated
...
deployment "notes" successfully rolled out
code = NotFound means: registry reached, tag missing.
With a wrong registry address, you would see
connection refused or no such host there.
Save your progress? Sign up for free
Round 2: The key has a different name
Situation: Team 2 neatly puts the API key into a Secret. The Pod still keeps restarting.
k create ns tag18-r2
k config set-context --current --namespace=tag18-r2
k apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: notes-key
stringData:
API_KEY: lab-key-123
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 1
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
envFrom:
- secretRef:
name: notes-key
EOFTarget time: 3 min
Hint: envFrom takes the Secret’s keys
as variable names – exactly as they are written there.
Solution
k get pods
k logs deploy/notes
k logs deploy/notes --previous
k get secret notes-key -o jsonpath='{.data}'; echoExpected (abridged):
NAME READY STATUS RESTARTS AGE
notes-b675577f9-5qbdk 0/1 Error 2 (37s ago) 41s
time=2026-11-18T16:13:48.932Z level=ERROR msg="start abgebrochen" err="NOTES_API_KEY
oder NOTES_API_KEY_FILE fehlt"
unable to retrieve container logs for containerd://222bba0c36db…
{"API_KEY":"bGFiLWtleS0xMjM="}
Warning: In the lab (Kubernetes 1.37.2, containerd 2.4.2),
--previousonly returnsunable to retrieve container logsfor a Pod stuck inCrashLoopBackOff.k logswithout--previousshows the last finished run – that is enough.
The Secret provides API_KEY, the app wants
NOTES_API_KEY. Instead of changing the Secret (other teams
might use it), --prefix maps the name:
k set env deploy/notes --from=secret/notes-key --prefix=NOTES_
k rollout status deploy/notes
k set env deploy/notes --listExpected (abridged):
deployment.apps/notes env updated
deployment "notes" successfully rolled out
# Deployment notes, container notes-api
# NOTES_API_KEY from secret notes-key, key API_KEY
k set env --from writes one secretKeyRef
entry per key into the Deployment.
Save your progress? Sign up for free
Round 3: The ConfigMap is missing
Situation: Team 3 has split port and key: the port
via the ConfigMap notes-config, the key directly. The Pod
never gets past startup.
k create ns tag18-r3
k config set-context --current --namespace=tag18-r3
k apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: notes-config
data:
PORT: "8080"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 1
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
envFrom:
- configMapRef:
name: notes-conf
EOFTarget time: 2 min
Hint: Compare the name in the error with
k get cm.
Solution
k get pods
k describe pod -l app=notes | tail -1
k get cmExpected (abridged):
NAME READY STATUS RESTARTS AGE
notes-6b679fcc6b-nzfkl 0/1 CreateContainerConfigError 0 20s
Warning Failed 6s (x4 over 19s) kubelet spec.containers{notes-api}: Error:
configmap "notes-conf" not found
NAME DATA AGE
kube-root-ca.crt 1 22s
notes-config 1 21s
In the exam, k edit deploy notes and
notes-conf → notes-config is enough. Without
an editor, use a JSON patch (lists without a merge key, such as
envFrom, can only be changed precisely this way):
k patch deploy notes --type=json -p='[{"op": "replace",
"path": "/spec/template/spec/containers/0/envFrom/0/configMapRef/name",
"value": "notes-config"}]'
k rollout status deploy/notesExpected (abridged):
deployment.apps/notes patched
deployment "notes" successfully rolled out
A missing key in an existing ConfigMap
(configMapKeyRef) also ends in
CreateContainerConfigError, just with
couldn't find key.
Save your progress? Sign up for free
Round 4: The Service is silent
Situation: Team 4 has a Deployment and a Service.
The test Pod client gets no answer from notes.
Watch out: there are two errors.
k create ns tag18-r4
k config set-context --current --namespace=tag18-r4
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 2
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /healthz
port: 8081
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: notes
spec:
selector:
app: notes-api
ports:
- port: 80
targetPort: 8080
EOF
k run client --image=busybox:1.37 -- sleep 3600Target time: 4 min
Hint: An endpoint needs two things: a matching label
and ready=true.
Solution
k get pods
k exec client -- wget -qO- -T3 notes/healthz
k get endpointslices -l kubernetes.io/service-name=notes
k get svc notes -o wide
k get pods -l app=notes --show-labelsExpected (abridged):
NAME READY STATUS RESTARTS AGE
client 1/1 Running 0 30s
notes-548bf6d74d-256fw 0/1 Running 0 31s
notes-548bf6d74d-5p4p8 0/1 Running 0 31s
wget: can't connect to remote host (10.107.39.198): Connection refused
command terminated with exit code 1
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
notes-46zvl IPv4 <unset> <unset> 33s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
notes ClusterIP 10.107.39.198 <none> 80/TCP 33s app=notes-api
NAME READY STATUS RESTARTS AGE LABELS
notes-548bf6d74d-256fw 0/1 Running 0 33s app=notes,pod-template-...
Error 1: the selector looks for app=notes-api, the Pods
are labeled app=notes.
k set selector svc notes app=notes
k get endpointslices -l kubernetes.io/service-name=notes \
-o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]} '\
'{.conditions.ready}{"\n"}{end}'
k exec client -- wget -qO- -T3 notes/healthzExpected:
service/notes selector updated
192.168.235.190 false
192.168.189.79 false
wget: can't connect to remote host (10.107.39.198): Connection refused
command terminated with exit code 1
Error 2: the Pods are now in the slice, but not ready –
0/1 Running was the second clue.
k get events --field-selector reason=Unhealthy | tail -1
k patch deploy notes --type=json -p='[{"op": "replace",
"path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port",
"value": 8080}]'
k rollout status deploy/notes
k exec client -- wget -qO- -T3 notes/healthz; echoExpected (abridged):
4s Warning Unhealthy pod/notes-548bf6d74d-5p4p8 Readiness probe failed: Get
"http://192.168.189.79:8081/healthz": dial tcp 192.168.189.79:8081: connect: connection
refused
deployment.apps/notes patched
deployment "notes" successfully rolled out
ok
With Calico, Connection refused at the ClusterIP means:
a Service without ready endpoints. A timeout points more toward a
NetworkPolicy (round 8).
Save your progress? Sign up for free
Round 5: The export dies silently
Situation: Team 5 runs an export job as a
Deployment. It keeps restarting, and k logs stays
empty.
k create ns tag18-r5
k config set-context --current --namespace=tag18-r5
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: export
spec:
replicas: 1
selector:
matchLabels:
app: export
template:
metadata:
labels:
app: export
spec:
containers:
- name: export
image: busybox:1.37
command: ["sh", "-c", "head -c 80m /dev/zero | tail > /dev/null;
echo export fertig; sleep 3600"]
resources:
requests:
memory: 32Mi
limits:
memory: 32Mi
EOFTarget time: 2 min
Hint: Empty logs plus restarts – the reason is in the status of the last run.
Solution
k get pods
k logs deploy/export
k describe pod -l app=export | grep -A3 "Last State"Expected (abridged):
NAME READY STATUS RESTARTS AGE
export-b689db759-p7c8t 0/1 OOMKilled 2 (25s ago) 30s
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
tail keeps the 80 MiB without a line ending in memory,
while the limit allows 32 MiB. The kernel kills the container before it
writes a single line.
k set resources deploy export --requests=memory=128Mi --limits=memory=128Mi
k rollout status deploy/export
k logs deploy/exportExpected (abridged):
deployment.apps/export resource requirements updated
deployment "export" successfully rolled out
export fertig
Save your progress? Sign up for free
Round 6: No Pods, no error on the Deployment
Situation: Team 6 does not see a single Pod in its
new namespace. k get pods says
No resources found.
k create ns tag18-r6
k config set-context --current --namespace=tag18-r6
k apply -f - <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata:
name: team
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.memory: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 3
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
EOFTarget time: 3 min
Hint: Pods are created by the ReplicaSet – that is where the events are.
Solution
k get deploy,rs
k get events --field-selector reason=FailedCreate | tail -1
k describe quota teamExpected (abridged):
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/notes 0/3 0 0 10s
NAME DESIRED CURRENT READY AGE
replicaset.apps/notes-5fdf9f698c 3 0 0 10s
1s Warning FailedCreate replicaset/notes-5fdf9f698c (combined from similar
events): Error creating: pods "notes-5fdf9f698c-4s7hm" is forbidden: failed quota: team:
must specify limits.memory for: notes-api; requests.cpu for: notes-api; requests.memory
for: notes-api
Resource Used Hard
-------- ---- ----
limits.memory 0 1Gi
requests.cpu 0 1
requests.memory 0 1Gi
k set resources deploy notes --requests=cpu=100m,memory=64Mi --limits=memory=64Mi
k rollout status deploy/notes
k describe quota team | tail -3Expected (abridged):
deployment.apps/notes resource requirements updated
deployment "notes" successfully rolled out
limits.memory 192Mi 1Gi
requests.cpu 300m 1
requests.memory 192Mi 1Gi
DESIRED 3, CURRENT 0 is the telltale sign: the
controller wants Pods, the API server refuses. A LimitRange with default
values (Day 7) would have prevented the error.
Save your progress? Sign up for free
Round 7: The sidecar isn’t allowed
Situation: Team 7 has hardened notes-api with
runAsNonRoot and added a small health poller. The Pod stays
at 1/2.
k create ns tag18-r7
k config set-context --current --namespace=tag18-r7
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 1
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
securityContext:
runAsNonRoot: true
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
- name: poller
image: busybox:1.37
command: ["sh", "-c", "while true; do
wget -qO- localhost:8080/healthz; echo; sleep 10; done"]
EOFTarget time: 3 min
Hint: runAsNonRoot forbids root, but
does not set a user by itself.
Solution
k get pods
k get events --field-selector type=Warning | tail -1Expected (abridged):
NAME READY STATUS RESTARTS AGE
notes-546897878f-fqkzn 1/2 CreateContainerConfigError 0 20s
5s Warning Failed pod/notes-546897878f-fqkzn Error: container has runAsNonRoot
and image will run as root (pod: "notes-546897878f-fqkzn_tag18-r7(9331…)", container:
poller)
notes-api runs as 65532 and passes the check, busybox runs as root. The hardening stays; the poller gets its own user:
k patch deploy notes --type=json -p='[{"op": "add",
"path": "/spec/template/spec/containers/1/securityContext",
"value": {"runAsUser": 65532}}]'
k rollout status deploy/notes
k logs deploy/notes -c pollerExpected (abridged):
deployment.apps/notes patched
deployment "notes" successfully rolled out
ok
Removing runAsNonRoot would be wrong – in the exam, you
would lose the points for the security requirement.
Save your progress? Sign up for free
Round 8: Nothing gets in from outside
Situation: Team 8 publishes notes-api through
Traefik as notes.tag18.example. The namespace has a
default-deny policy. Nothing gets in from outside – again
two errors.
k create ns tag18-r8
k config set-context --current --namespace=tag18-r8
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 2
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: notes
spec:
selector:
app: notes
ports:
- name: http
port: 80
targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: notes
spec:
ingressClassName: traefik
rules:
- host: notes.tag18.example
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: notes
port:
number: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
EOFTarget time: 5 min
Hint: The Ingress points to the Service port, not to the container port. After that: from which namespace does Traefik’s traffic come?
Solution
curl -s -H 'Host: notes.tag18.example' http://172.16.0.126:30080/healthz
k describe ingress notes | grep -A3 Rules
k get svc notes
k logs -n traefik deploy/traefik --since=60s | grep tag18 | tail -1Expected (abridged, color codes removed):
404 page not found
Rules:
Host Path Backends
---- ---- --------
notes.tag18.example
/ notes:8080 ()
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
notes ClusterIP 10.108.142.39 <none> 80/TCP 20s
… ERR Cannot create service error="service port not found" ingress=notes
namespace=tag18-r8 … serviceName=notes
servicePort=&ServiceBackendPort{Name:,Number:8080,}
Empty parentheses after the backend: the Service has no port 8080. Fix error 1:
k patch ingress notes --type=json -p='[{"op": "replace",
"path": "/spec/rules/0/http/paths/0/backend/service/port/number", "value": 80}]'
k describe ingress notes | grep ' /'
curl -s -w '\n%{http_code}\n' -H 'Host: notes.tag18.example' \
http://172.16.0.126:30080/healthzExpected (the answer takes 30 s):
ingress.networking.k8s.io/notes patched
/ notes:80 (192.168.189.76:8080,192.168.235.165:8080)
Gateway Timeout
504
Error 2: Traefik cannot reach the Pods – default-deny
blocks all incoming traffic. You specifically allow the namespace
traefik on port 8080 (Pod port, not Service port):
k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: notes-from-traefik
spec:
podSelector:
matchLabels:
app: notes
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: traefik
ports:
- port: 8080
EOF
curl -s -H 'Host: notes.tag18.example' -H 'X-API-Key: lab-key-123' \
-d '{"text":"gewonnen"}' http://172.16.0.126:30080/notes; echoExpected:
networkpolicy.networking.k8s.io/notes-from-traefik created
{"id":"4d28a99e-ba4b-4613-8da2-43d02bd1501e","text":"gewonnen"}
404 from Traefik means: no matching route.
504 means: the route is there, but the backend does not
answer – typical for a NetworkPolicy.
Stop the clock.
Save your progress? Sign up for free
Scoring
| Round | Failure pattern | Target time | Points |
|---|---|---|---|
| 1 | ImagePullBackOff (tag) | 2 min | 1 |
| 2 | CrashLoop, variable has a different name | 3 min | 2 |
| 3 | CreateContainerConfigError (ConfigMap) | 2 min | 1 |
| 4 | Selector + readiness port | 4 min | 2 |
| 5 | OOMKilled | 2 min | 1 |
| 6 | Quota, FailedCreate | 3 min | 2 |
| 7 | runAsNonRoot vs. root image | 3 min | 2 |
| 8 | Ingress port + NetworkPolicy | 5 min | 3 |
You only get full points within the target time and without a hint; with a hint or over time, half (round down). Overall target time: 24 min, maximum: 14 points.
| Points | Assessment |
|---|---|
| 12–14 | exam-ready – move on to the Practice Exam (Appendix B) |
| 8–11 | good; repeat the days of the rounds you missed |
| 0–7 | work through Days 14 and 15 once more |
Checkpoint
k get deploy -A | grep tag18Expected: eight lines from tag18-r1 to
tag18-r8, each with all replicas ready (2/2,
1/1, 1/1, 2/2, 1/1,
3/3, 1/1, 2/2).
k get pods -A --no-headers | grep tag18 | grep -vc RunningExpected: 0
k exec -n tag18-r4 client -- wget -qO- -T3 notes/healthz; echoExpected: ok
curl -s -H 'Host: notes.tag18.example' http://172.16.0.126:30080/version; echoExpected: 1.0
Challenge
Build your own round for a study partner with two errors from two
different days, write down the situation in two sentences and let them
play. Good candidates: a liveness probe on the wrong path (Day 14), a
secretKeyRef with the wrong key (Day 6), a
rollout with maxUnavailable: 0 and a broken image (Day
11).
Hint: Test the round yourself first and note down the proof command.
Solution
There is no fixed solution. Afterwards, check together with the checkpoint commands whether everything is running again.
Clean up:
k config set-context --current --namespace=default
k delete ns tag18-r1 tag18-r2 tag18-r3 tag18-r4 tag18-r5 tag18-r6 tag18-r7 tag18-r8Save your progress? Sign up for free
Exam Tip
- Always proceed the same way:
k get→ events/describe→logs. The status column almost always tells you which of the three tools has the answer. - Finish every task with a proof command – that is the only way you notice the second error (rounds 4 and 8).
- Fix the error in the existing object, don’t reinvent the object: names, labels and security requirements from the task stay as they are.
Enterprise Note
- Most of today’s errors are caught by a CI check before deploying:
kubeconformfor the schema,kube-linteror policies (Kyverno, OPA) for probes, resources andsecurityContext. - Alerts on
kube_deployment_status_replicas_unavailableandkube_pod_container_status_last_terminated_reason{reason="OOMKilled"}report broken Deployments before a user does. - Pin image tags in production by digest; a typo in the tag then already shows up when the pipeline resolves it.