Day 20: Competition – the broken cluster
Goal: In eight rounds you find and fix planted faults from across the book – against the clock, with proof before and after. · Duration: approx. 120 min · Prerequisite: Days 0–19
Briefing
- In
tag20you build thenotesapp:web(nginx) forwards/api/tonotes-api,notes-apistores its data inpostgres(local-path volume). In front of it sits Traefik with the hostnotes.tag20.lab. - Eight rounds, all layers: quota, Service, admission, network, Ingress, storage, webhook, node. The rounds build on each other: play them in order.
- Tools: everything from the book –
get,describe,events,logs, k9s,get --raw, nodes via SSH. - Every round ends with the measurement: broken before, healthy after.
curl ──> Traefik (Host notes.tag20.lab) ──> web ──/api/──> notes-api ──> postgres
(PVC)
Round Symptom Target time
1 New version of notes-api does not arrive 4 min
2 503 on everything 3 min
3 Rollout hangs, Pod does not start 5 min
4 notes-api does not start after a restart 5 min
5 404 page not found 3 min
6 Requests hang, postgres Pending 5 min
7 web cannot be rolled out again 4 min
8 Maintenance: drain hangs 4 min
Rules of the Game
- In each round, the referee runs the referee
commands on
cp1– a colleague, or you yourself without looking: copy the block, paste it into a second terminal, close the window. Then start the clock. - Find the cause, fix it, prove it with the measurement. Only then stop the clock.
- Fix the cause, not the symptom: if you rebuild the namespace, delete policies, quota or PDB instead of correcting them, switch off Kyverno or recreate the database, you get zero points. The note “Wettkampf” (competition) must still be in the database at the end.
- Read the hint only after the target time has run out.
- Pod names and hashes in the outputs will differ on your side; the
commands therefore work with
-l app=…ordeploy/…. - The goal of every round is the same clean end state. The command for it is called measurement from here on:
# on cp1
U='--resolve notes.tag20.lab:80:172.16.0.126 http://notes.tag20.lab'
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasHealthy means: the note Wettkampf as JSON, all Pods
ready and up to date (READY equal to AKTUELL
(current): 2 for notes-api and web, 1 for
postgres). Right after a rollout, nginx answers with 502
for a few seconds – that is why the solutions put a short
sleep before the measurement.
Playing Field
The app is the same as in “Linkerd hands-on” and “CKAD hands-on”, here with a volume from local-path (Day 10) and Ingress via Traefik (Day 9). The values are lab values.
# on cp1
mkdir -p ~/debug/tag20 && cd ~/debug/tag20
kubectl create namespace tag20
kubectl -n tag20 create secret generic notes-db \
--from-literal=password=lab-db-123 \
--from-literal=url='postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable'
kubectl -n tag20 create secret generic notes-api-key --from-literal=key=lab-key-123
cat > postgres.yaml <<'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: postgres, namespace: tag20}
spec:
serviceName: postgres
replicas: 1
selector: {matchLabels: {app: postgres}}
template:
metadata: {labels: {app: postgres}}
spec:
containers:
- name: postgres
image: docker.io/library/postgres:18.7-alpine
env:
- {name: POSTGRES_USER, value: notes}
- {name: POSTGRES_DB, value: notes}
- name: POSTGRES_PASSWORD
valueFrom: {secretKeyRef: {name: notes-db, key: password}}
- {name: PGDATA, value: /var/lib/postgresql/data/pgdata}
ports: [{containerPort: 5432, name: postgres}]
readinessProbe:
exec: {command: [pg_isready, -U, notes, -d, notes]}
volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
volumeClaimTemplates:
- metadata: {name: data}
spec:
storageClassName: local-path
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Service
metadata: {name: postgres, namespace: tag20}
spec:
selector: {app: postgres}
ports: [{port: 5432, targetPort: postgres, name: postgres}]
EOF
cat > notes-api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: tag20}
spec:
replicas: 2
selector: {matchLabels: {app: notes-api}}
template:
metadata: {labels: {app: notes-api}}
spec:
initContainers:
- name: migrate
image: ghcr.io/thomaszachmann/notes-api:0.7.1
args: [migrate]
env:
- name: NOTES_DB_URL
valueFrom: {secretKeyRef: {name: notes-db, key: url}}
containers:
- name: notes-api
image: ghcr.io/thomaszachmann/notes-api:0.7.1
ports: [{containerPort: 8080, name: http}]
env:
- name: NOTES_DB_URL
valueFrom: {secretKeyRef: {name: notes-db, key: url}}
- name: NOTES_API_KEY
valueFrom: {secretKeyRef: {name: notes-api-key, key: key}}
readinessProbe:
httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag20}
spec:
selector: {app: notes-api}
ports: [{port: 80, targetPort: http, name: http}]
EOF
cat > web.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag20}
data:
default.conf: |
server {
listen 8080;
location = / { default_type text/plain; return 200 "notes web\n"; }
location /api/ {
proxy_set_header X-API-Key lab-key-123;
proxy_pass http://notes-api.tag20.svc.cluster.local/;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag20}
spec:
replicas: 2
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
containers:
- name: web
image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /, port: http}
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag20}
spec:
selector: {app: web}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: notes, namespace: tag20}
spec:
ingressClassName: traefik
rules:
- host: notes.tag20.lab
http:
paths:
- path: /
pathType: Prefix
backend: {service: {name: web, port: {number: 80}}}
EOF
kubectl apply -f postgres.yaml -f notes-api.yaml -f web.yaml
kubectl -n tag20 rollout status statefulset/postgres --timeout=180s
kubectl -n tag20 rollout status deploy/notes-api --timeout=180s | tail -1
kubectl -n tag20 rollout status deploy/web --timeout=180sExpected (abridged):
namespace/tag20 created
secret/notes-db created
secret/notes-api-key created
statefulset.apps/postgres created
...
ingress.networking.k8s.io/notes created
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...
deployment "notes-api" successfully rolled out
deployment "web" successfully rolled out
Create a note, then take the first measurement:
# on cp1
U='--resolve notes.tag20.lab:80:172.16.0.126 http://notes.tag20.lab'
curl -s -m 10 $U/api/notes -d '{"text":"Wettkampf"}'; echo
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasExpected:
{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
This is the healthy playing field. The manifests in
~/debug/tag20 are the desired state – comparing them with
the live object is allowed and often the fastest way.
Round 1: The new version
Situation: The team has rolled out
notes-api again. The pipeline has been waiting on
rollout status for minutes.
Referee commands:
# on cp1
kubectl -n tag20 create quota grenze --hard=pods=5
kubectl -n tag20 rollout restart deploy/notes-apiTarget time: 4 min
Hint: The new Pod is not missing because of the Pod – ask the ReplicaSet.
Solution Round 1
# on cp1
sleep 20
kubectl -n tag20 rollout status deploy/notes-api --timeout=5s
kubectl -n tag20 get rs -l app=notes-api
kubectl -n tag20 events --types=Warning | grep FailedCreate | tail -1 | cut -c1-260 \
| fold -w 86
kubectl -n tag20 describe quota grenze | tail -3Expected (line wrapped):
Waiting for deployment "notes-api" rollout to finish: 0 out of 2 new replicas have been
updated...
error: timed out waiting for the condition
NAME DESIRED CURRENT READY AGE
notes-api-6c5f9d64bc 1 0 0 25s
notes-api-6d87674cb5 2 2 2 73s
6s (x4 over 24s) Warning FailedCreate ReplicaSet/notes-api-6c5f9d64bc (combin
ed from similar events): Error creating: pods "notes-api-6c5f9d64bc-8bxpv" is forbidde
n: exceeded quota: grenze, requested: pods=1, used: pods=5, limited: pods=5
Resource Used Hard
-------- ---- ----
pods 5 5
The quota allows exactly as many Pods as are running. A rollout
briefly needs one more Pod (maxSurge), and the quota
rejects it at the ReplicaSet (Day 3). The quota stays – just with room
for rollouts:
# on cp1
kubectl -n tag20 patch resourcequota grenze -p '{"spec":{"hard":{"pods":"8"}}}'
kubectl -n tag20 rollout status deploy/notes-api --timeout=120s | tail -1Expected:
resourcequota/grenze patched
deployment "notes-api" successfully rolled out
Measurement (commands from the rules of the game):
# on cp1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasExpected:
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
Save your progress? Sign up for free
Round 2: 503
Situation: “I only made the labels consistent.” Since then the site answers with 503 – even the start page.
Referee commands:
# on cp1
kubectl -n tag20 patch service web -p '{"spec":{"selector":{"app":"wbe"}}}'Target time: 3 min
Hint: Who delivers the 503 – nginx or Traefik? And where does Traefik want to go?
Solution Round 2
# on cp1
sleep 10
curl -s -m 10 -w ' %{http_code}\n' $U/
kubectl -n tag20 get endpointslices -l kubernetes.io/service-name=web
kubectl -n tag20 get svc web -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n tag20 get pods -l app=web --no-headers | wc -lExpected:
no available server
503
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
web-hbj62 IPv4 <unset> <unset> 113s
{"app":"wbe"}
2
no available server is the answer from Traefik itself:
the Service has no endpoints because its selector matches no Pod (Day
7). Back to the desired value from web.yaml:
# on cp1
kubectl -n tag20 patch service web -p '{"spec":{"selector":{"app":"web"}}}'
sleep 3
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasExpected:
service/web patched
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
Save your progress? Sign up for free
Round 3: The unified policy
Situation: The platform team has rolled out a
Kyverno policy “to unify the API keys”. Afterwards,
notes-api was restarted – and is stuck.
Referee commands:
# on cp1
cat > /tmp/mpol.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata: {name: api-key-vereinheitlichen}
spec:
matchConstraints:
namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag20}
objectSelector:
matchLabels: {app: notes-api}
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE]
resources: [pods]
mutations:
- patchType: JSONPatch
jsonPatch:
expression: >-
[JSONPatch{op: "replace",
path: "/spec/containers/0/env/1/valueFrom/secretKeyRef/key",
value: "api-key"}]
EOF
kubectl apply -f /tmp/mpol.yaml && rm /tmp/mpol.yaml
sleep 5
kubectl -n tag20 rollout restart deploy/notes-apiTarget time: 5 min
Hint: Compare the Pod template in the Deployment with the Pod that is stuck.
Solution Round 3
# on cp1
sleep 30
kubectl -n tag20 get pods -l app=notes-api
kubectl -n tag20 events --types=Warning | grep -m1 "couldn't find key" \
| grep -o 'Error:.*'
J='{.spec.template.spec.containers[0].env[1].valueFrom.secretKeyRef}{"\n"}'
kubectl -n tag20 get deploy notes-api -o jsonpath="$J"
P=$(kubectl -n tag20 get pods -l app=notes-api --no-headers | grep -v Running \
| awk '{print $1}')
kubectl -n tag20 get pod $P \
-o jsonpath='{.spec.containers[0].env[1].valueFrom.secretKeyRef}{"\n"}'
kubectl get mpolExpected:
NAME READY STATUS RESTARTS AGE
notes-api-67f8dc996-6m9fz 0/1 CreateContainerConfigError 0 30s
notes-api-6c5f9d64bc-5p7k7 1/1 Running 0 64s
notes-api-6c5f9d64bc-ffm7k 1/1 Running 0 60s
Error: couldn't find key api-key in Secret tag20/notes-api-key
{"key":"key","name":"notes-api-key"}
{"key":"api-key","name":"notes-api-key"}
NAME AGE READY
api-key-vereinheitlichen 37s true
CreateContainerConfigError with a missing key (Day 4) –
but the Deployment asks for the correct key key. The Pod
only got the wrong one when it was created: a mutation by Kyverno (Day
12). That is why the manifest looks clean, only the live object does
not. The policy is the cause; it asks for a key that the Secret does not
have. Remove it (or correct it with the platform team), then have the
stuck Pod recreated:
# on cp1
kubectl delete mpol api-key-vereinheitlichen
kubectl -n tag20 delete pod $P
kubectl -n tag20 rollout status deploy/notes-api --timeout=120s | tail -1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasExpected:
mutatingpolicy.policies.kyverno.io "api-key-vereinheitlichen" deleted
pod "notes-api-67f8dc996-6m9fz" deleted from tag20 namespace
deployment "notes-api" successfully rolled out
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
Save your progress? Sign up for free
Round 4: Only the database
Situation: Security has restricted
notes-api “to the bare minimum”. It still works. Only after
the next restart does someone call.
Referee commands:
# on cp1
cat > /tmp/np.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: nur-datenbank, namespace: tag20}
spec:
podSelector: {matchLabels: {app: notes-api}}
policyTypes: [Egress]
egress:
- to:
- podSelector: {matchLabels: {app: postgres}}
ports: [{port: 5432, protocol: TCP}]
EOF
kubectl apply -f /tmp/np.yaml && rm /tmp/np.yaml
kubectl -n tag20 rollout restart deploy/notes-apiTarget time: 5 min
Hint: Read the log of the init container – which address does it want to reach?
Solution Round 4
# on cp1
sleep 40
kubectl -n tag20 get pods -l app=notes-api
P=$(kubectl -n tag20 get pods -l app=notes-api --no-headers | grep Init \
| awk '{print $1}')
kubectl -n tag20 logs $P -c migrate --tail 1 | cut -c1-260 | fold -w 86
kubectl -n tag20 get networkpolicyExpected:
NAME READY STATUS RESTARTS AGE
notes-api-6464cd5898-gl95n 0/1 Init:Error 2 (32s ago) 40s
notes-api-67f8dc996-7vfx7 1/1 Running 0 50s
notes-api-67f8dc996-ljg6g 1/1 Running 0 54s
time=2026-11-20T14:13:35.067Z level=ERROR msg="database unreachable" source=env err="f
ailed to connect to `user=notes database=notes`: hostname resolving error: lookup post
gres on 10.43.0.10:53: dial udp 10.43.0.10:53: i/o timeout"
NAME POD-SELECTOR AGE
nur-datenbank app=notes-api 42s
The database is allowed, but not the way to its
name: 10.43.0.10:53 is CoreDNS (Day 8).
The old Pods kept running because their connections were already
established – only a restart has to resolve the name again. The policy
stays, it just gets DNS added:
# on cp1
cat > np-fix.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: nur-datenbank, namespace: tag20}
spec:
podSelector: {matchLabels: {app: notes-api}}
policyTypes: [Egress]
egress:
- to:
- podSelector: {matchLabels: {app: postgres}}
ports: [{port: 5432, protocol: TCP}]
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
podSelector: {matchLabels: {k8s-app: kube-dns}}
ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]
EOF
kubectl apply -f np-fix.yaml
kubectl -n tag20 rollout status deploy/notes-api --timeout=180s | tail -1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasExpected:
networkpolicy.networking.k8s.io/nur-datenbank configured
deployment "notes-api" successfully rolled out
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
The init container is stuck in backoff (Day 5); it can take up to a minute until it tries again.
Save your progress? Sign up for free
Round 5: 404
Situation: “We are moving to ingress-nginx soon, I already switched the Ingress over.”
Referee commands:
# on cp1
kubectl -n tag20 patch ingress notes --type=merge \
-p '{"spec":{"ingressClassName":"nginx"}}'Target time: 3 min
Hint: Which IngressClasses exist in the cluster?
Solution Round 5
# on cp1
sleep 5
curl -s -m 10 -w ' %{http_code}\n' $U/api/notes
kubectl -n tag20 get ingress notes
kubectl get ingressclassExpected:
404 page not found
404
NAME CLASS HOSTS ADDRESS PORTS AGE
notes nginx notes.tag20.lab 80 4m6s
NAME CONTROLLER PARAMETERS AGE
traefik (default) traefik.io/ingress-controller <none> 7h41m
404 page not found is Traefik’s answer for “no route”:
it silently ignores an Ingress with a foreign class, without an event
(Day 9). There is no controller for nginx:
# on cp1
kubectl -n tag20 patch ingress notes --type=merge \
-p '{"spec":{"ingressClassName":"traefik"}}'
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasExpected:
ingress.networking.k8s.io/notes patched
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
Save your progress? Sign up for free
Round 6: The database moves
Situation: “Postgres is supposed to go to the node
with the faster disk.” Since then, requests to notes hang
until the timeout. The referee picks a worker on which Postgres is
currently not running.
Referee commands:
# on cp1
N=$(kubectl -n tag20 get pod postgres-0 -o jsonpath='{.spec.nodeName}')
A=$(kubectl get nodes -o name | cut -d/ -f2 | grep -v -x -e cp1 -e $N | head -1)
kubectl -n tag20 patch statefulset postgres \
-p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"'$A'"}}}}}'
kubectl -n tag20 delete pod postgres-0Target time: 5 min
Hint: Where is the volume? And does a StatefulSet replace a Pod that never starts on its own?
Solution Round 6
# on cp1
sleep 20
kubectl -n tag20 get pod postgres-0
kubectl -n tag20 events --for pod/postgres-0 | tail -1 | cut -c1-260 | fold -w 86
kubectl -n tag20 get pod postgres-0 -o jsonpath='{.spec.nodeSelector}{"\n"}'
PV=$(kubectl -n tag20 get pvc data-postgres-0 -o jsonpath='{.spec.volumeName}')
kubectl get pv $PV -o jsonpath='{.spec.nodeAffinity.required}{"\n"}' | fold -w 86Expected:
NAME READY STATUS RESTARTS AGE
postgres-0 0/1 Pending 0 20s
20s (x2 over 20s) Warning FailedScheduling Pod/postgres-0 0/3 nodes are av
ailable: 1 node(s) didn't match PersistentVolume's node affinity, 2 node(s) didn't mat
ch Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is
no
{"kubernetes.io/hostname":"worker2"}
{"nodeSelectorTerms":[{"matchExpressions":[{"key":"kubernetes.io/hostname","operator":
"In","values":["worker1"]}]}]}
The local-path volume lives on a specific node and does not move
along (Day 10); the nodeSelector points to a different one.
notes-api waits for the database, nginx gives up with 504
after 60 seconds. First remove the selector – and then delete the stuck
Pod: a StatefulSet does not replace a Pod with the old template as long
as that Pod was never ready (the Kubernetes docs call this “forced
rollback”).
# on cp1
kubectl -n tag20 patch statefulset postgres --type=json \
-p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
kubectl -n tag20 delete pod postgres-0
kubectl -n tag20 rollout status statefulset/postgres --timeout=120s | tail -1
sleep 15
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasExpected:
statefulset.apps/postgres patched
pod "postgres-0" deleted from tag20 namespace
partitioned roll out complete: 1 new pods have been updated...
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
The note is still there – the data was safe on the old node the whole time.
Save your progress? Sign up for free
Round 7: The validator
Situation: A vendor tool was installed “for testing”
and removed again. Now web is supposed to get a new
configuration, but no new Pod is created.
Referee commands:
# on cp1
cat > /tmp/wh.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: {name: tag20-pruefer}
webhooks:
- name: pruefer.tag20.lab
admissionReviewVersions: [v1]
sideEffects: None
failurePolicy: Fail
namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag20}
rules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE]
resources: [pods]
clientConfig:
service: {name: pruefer, namespace: tag20, path: /validate}
EOF
kubectl apply -f /tmp/wh.yaml && rm /tmp/wh.yaml
kubectl -n tag20 rollout restart deploy/webTarget time: 4 min
Hint: Internal error occurred means:
the rejection did not come from the API server itself.
Solution Round 7
# on cp1
sleep 15
kubectl -n tag20 get deploy web
kubectl -n tag20 events --types=Warning | grep -m1 'failed calling webhook' \
| cut -c1-280 | fold -w 86
kubectl get validatingwebhookconfigurations -o name | grep -v kyverno
kubectl -n tag20 get svc prueferExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
web 2/2 0 2 5m12s
5s (x12 over 15s) Warning FailedCreate ReplicaSet/web-5489c7bf58
Error creating: Internal error occurred: failed calling webhook "pruefer.tag20.lab":
failed to call webhook: Post "https://pruefer.tag20.svc:443/validate?timeout=10s": ser
vice "pruefer" not fou
validatingwebhookconfiguration.admissionregistration.k8s.io/tag20-pruefer
Error from server (NotFound): services "pruefer" not found
A webhook without a Service, failurePolicy: Fail, valid
for every Pod in tag20 (Day 16). The tool is gone, its
configuration is not – so it has to go as well:
# on cp1
kubectl delete validatingwebhookconfiguration tag20-pruefer
kubectl -n tag20 rollout status deploy/web --timeout=120s | tail -1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicasExpected:
validatingwebhookconfiguration.admissionregistration.k8s.io "tag20-pruefer" deleted
deployment "web" successfully rolled out
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
Save your progress? Sign up for free
Round 8: Maintenance window
Situation: A worker is to be patched. The referee
picks the worker without Postgres and starts the drain with a 40-second
deadline. Then it is your turn: the node is to be emptied,
notes is to stay reachable, and at the end the node is back
in service.
Referee commands:
# on cp1
cat > /tmp/pdb.yaml <<'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: notes-pdb, namespace: tag20}
spec:
maxUnavailable: 0
selector:
matchExpressions:
- {key: app, operator: In, values: [web, notes-api]}
EOF
kubectl apply -f /tmp/pdb.yaml && rm /tmp/pdb.yaml
P=$(kubectl -n tag20 get pod postgres-0 -o jsonpath='{.spec.nodeName}')
W=$(kubectl get nodes -o name | cut -d/ -f2 | grep -v -x -e cp1 -e $P | head -1)
echo "Wartung: $W"
kubectl drain $W --ignore-daemonsets --delete-emptydir-data --timeout=40s 2>&1 \
| tail -1 | cut -c1-120Target time: 4 min
Hint: ALLOWED DISRUPTIONS.
Solution Round 8
W is the node the referee named – determined here with
the same rule:
# on cp1
P=$(kubectl -n tag20 get pod postgres-0 -o jsonpath='{.spec.nodeName}')
W=$(kubectl get nodes -o name | cut -d/ -f2 | grep -v -x -e cp1 -e $P | head -1)
echo $W
kubectl get nodes
kubectl -n tag20 get pdb
kubectl -n tag20 get pods -l 'app in (web,notes-api)' -o wide | awk '{print $1, $7}'Expected:
worker2
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane,etcd 7h45m v1.37.2+rke2r1
worker1 Ready <none> 7h42m v1.37.2+rke2r1
worker2 Ready,SchedulingDisabled <none> 7h41m v1.37.2+rke2r1
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
notes-pdb N/A 0 0 42s
NAME NODE
notes-api-6464cd5898-gl95n worker1
notes-api-6464cd5898-sjj98 worker2
web-5489c7bf58-9gncl worker2
web-5489c7bf58-vjtlf worker1
A PDB with maxUnavailable: 0 forbids every voluntary
eviction – the drain waits forever (Day 14). With two replicas, one may
go:
# on cp1
kubectl -n tag20 patch pdb notes-pdb --type=merge -p '{"spec":{"maxUnavailable":1}}'
kubectl drain $W --ignore-daemonsets --delete-emptydir-data --timeout=120s \
2>&1 | tail -1
kubectl uncordon $W
kubectl -n tag20 rollout status deploy/notes-api --timeout=120s | tail -1
kubectl -n tag20 rollout status deploy/web --timeout=120s | tail -1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas
kubectl get nodesExpected:
poddisruptionbudget.policy/notes-pdb patched
node/worker2 drained
node/worker2 uncordoned
deployment "notes-api" successfully rolled out
deployment "web" successfully rolled out
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME READY AKTUELL
notes-api 2 2
web 2 2
postgres 1 1
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane,etcd 7h45m v1.37.2+rke2r1
worker1 Ready <none> 7h42m v1.37.2+rke2r1
worker2 Ready <none> 7h41m v1.37.2+rke2r1
Save your progress? Sign up for free
Scoring
| Round | Fault | Target time | Points |
|---|---|---|---|
| 1 | Quota without room for maxSurge |
4 min | 2 |
| 2 | Service selector wbe |
3 min | 2 |
| 3 | MutatingPolicy sets the wrong Secret key | 5 min | 3 |
| 4 | Egress policy without DNS | 5 min | 2 |
| 5 | ingressClassName: nginx |
3 min | 1 |
| 6 | nodeSelector against local-path volume, Pod stuck |
5 min | 3 |
| 7 | orphaned webhook with failurePolicy: Fail |
4 min | 2 |
| 8 | PDB maxUnavailable: 0, node not put back into
service |
4 min | 3 |
Full points only within the target time, without the hint
and with the measurement; otherwise half (round down).
Zero points for symptom fixes: deleting the quota or PDB, removing the
policy instead of adding a DNS rule, shutting down Kyverno, recreating
the PVC or namespace. In round 8 you lose one point if the node is still
SchedulingDisabled at the end. Total target time: 33 min,
maximum: 18 points.
| Points | Assessment |
|---|---|
| 15–18 | ready for the final exam (Appendix B) |
| 10–14 | good; repeat the days of the rounds you missed |
| 0–9 | work through Days 3, 7, 8, 10, 12 and 14 again |
Challenge
Bonus round without points: a colleague started a Helm chart
info for notes, and the pipeline was aborted
in the middle of the upgrade. Bring the release back into a state in
which helm upgrade works – without touching the release
Secrets.
Referee commands:
# on cp1
cd ~/debug/tag20
helm create info >/dev/null
helm install info ./info -n tag20 --wait --timeout 2m | grep STATUS
helm upgrade info ./info -n tag20 --set image.tag=gibtsnicht --wait --timeout 5m &
sleep 5
kill -9 %1Hint: Day 17, Drill 3.
Solution
# on cp1
helm upgrade info ./info -n tag20 2>&1 | tail -1
helm -n tag20 history info -o json | jq -r '.[] | "\(.revision) \(.status)"'
helm -n tag20 rollback info 1 2>&1 | tail -1
helm upgrade info ./info -n tag20 --wait | grep -E 'STATUS|REVISION'Expected:
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
1 deployed
2 pending-upgrade
Rollback was a success! Happy Helming!
STATUS: deployed
REVISION: 4
pending-upgrade blocks every further upgrade; a
rollback to the last good revision releases the lock.
Save your progress? Sign up for free
Cleanup
# on cp1
helm uninstall info -n tag20
kubectl delete namespace tag20
kubectl get pv --no-headers | wc -lExpected:
release "info" uninstalled
namespace "tag20" deleted
No resources found
0
With the namespace, the PVC disappears, and local-path deletes the
volume (reclaimPolicy: Delete).
Checkpoint
# on cp1
kubectl get nodesExpected: three nodes Ready, none
SchedulingDisabled.
# on cp1
kubectl get mpol,vpol --no-headers 2>&1; kubectl get validatingwebhookconfigurations \
-o name | grep -c -v kyvernoExpected: No resources found and
0 – no leftovers from rounds 3 and 7.
Interview Note
- A rollout is stuck without an error on the Pod – where do you
look? At the ReplicaSet:
FailedCreatecaused by quota, an admission webhook or Pod Security shows up there, not on the Deployment. - The manifest is correct, but the Pod is still wrong?
Mutating admission (Kyverno, webhooks): compare the live object with the
manifest,
kubectl get mpolormutatingwebhookconfigurations. - Why does a StatefulSet Pod not restart after the fix? A Pod with the old template that was never ready is not replaced automatically – delete it, then the new template takes effect.
Enterprise Note
- Play game days like this one regularly with the on-call team; the solutions become runbooks.
- Always plan quotas with headroom for
maxSurgeand with an alert onFailedCreate. - Roll out mutating policies only with a review and a test run
(
kyverno test, Day 12); keep an eye on the policy reports. - Every tool that installs webhooks belongs in an inventory with an owner; find orphaned webhook configurations via policy or alert.
- PDBs belong in code review:
maxUnavailable: 0orminAvailableequal to the replica count blocks every maintenance.