Skip to content
Hands-on IT IT Drills

Day 20: Competition – the broken cluster

Goal: In eight rounds you find and fix planted faults from across the book – against the clock, with proof before and after. · Duration: approx. 120 min · Prerequisite: Days 0–19

Briefing

  • In tag20 you build the notes app: web (nginx) forwards /api/ to notes-api, notes-api stores its data in postgres (local-path volume). In front of it sits Traefik with the host notes.tag20.lab.
  • Eight rounds, all layers: quota, Service, admission, network, Ingress, storage, webhook, node. The rounds build on each other: play them in order.
  • Tools: everything from the book – get, describe, events, logs, k9s, get --raw, nodes via SSH.
  • Every round ends with the measurement: broken before, healthy after.
 curl ──> Traefik (Host notes.tag20.lab) ──> web ──/api/──> notes-api ──> postgres
                                                                          (PVC)
Round  Symptom                                         Target time
1      New version of notes-api does not arrive        4 min
2      503 on everything                               3 min
3      Rollout hangs, Pod does not start               5 min
4      notes-api does not start after a restart        5 min
5      404 page not found                              3 min
6      Requests hang, postgres Pending                 5 min
7      web cannot be rolled out again                  4 min
8      Maintenance: drain hangs                        4 min

Rules of the Game

  • In each round, the referee runs the referee commands on cp1 – a colleague, or you yourself without looking: copy the block, paste it into a second terminal, close the window. Then start the clock.
  • Find the cause, fix it, prove it with the measurement. Only then stop the clock.
  • Fix the cause, not the symptom: if you rebuild the namespace, delete policies, quota or PDB instead of correcting them, switch off Kyverno or recreate the database, you get zero points. The note “Wettkampf” (competition) must still be in the database at the end.
  • Read the hint only after the target time has run out.
  • Pod names and hashes in the outputs will differ on your side; the commands therefore work with -l app=… or deploy/….
  • The goal of every round is the same clean end state. The command for it is called measurement from here on:
# on cp1
U='--resolve notes.tag20.lab:80:172.16.0.126 http://notes.tag20.lab'
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Healthy means: the note Wettkampf as JSON, all Pods ready and up to date (READY equal to AKTUELL (current): 2 for notes-api and web, 1 for postgres). Right after a rollout, nginx answers with 502 for a few seconds – that is why the solutions put a short sleep before the measurement.

Playing Field

The app is the same as in “Linkerd hands-on” and “CKAD hands-on”, here with a volume from local-path (Day 10) and Ingress via Traefik (Day 9). The values are lab values.

# on cp1
mkdir -p ~/debug/tag20 && cd ~/debug/tag20
kubectl create namespace tag20
kubectl -n tag20 create secret generic notes-db \
  --from-literal=password=lab-db-123 \
  --from-literal=url='postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable'
kubectl -n tag20 create secret generic notes-api-key --from-literal=key=lab-key-123
cat > postgres.yaml <<'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: postgres, namespace: tag20}
spec:
  serviceName: postgres
  replicas: 1
  selector: {matchLabels: {app: postgres}}
  template:
    metadata: {labels: {app: postgres}}
    spec:
      containers:
      - name: postgres
        image: docker.io/library/postgres:18.7-alpine
        env:
        - {name: POSTGRES_USER, value: notes}
        - {name: POSTGRES_DB, value: notes}
        - name: POSTGRES_PASSWORD
          valueFrom: {secretKeyRef: {name: notes-db, key: password}}
        - {name: PGDATA, value: /var/lib/postgresql/data/pgdata}
        ports: [{containerPort: 5432, name: postgres}]
        readinessProbe:
          exec: {command: [pg_isready, -U, notes, -d, notes]}
        volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
  volumeClaimTemplates:
  - metadata: {name: data}
    spec:
      storageClassName: local-path
      accessModes: [ReadWriteOnce]
      resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Service
metadata: {name: postgres, namespace: tag20}
spec:
  selector: {app: postgres}
  ports: [{port: 5432, targetPort: postgres, name: postgres}]
EOF
cat > notes-api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: tag20}
spec:
  replicas: 2
  selector: {matchLabels: {app: notes-api}}
  template:
    metadata: {labels: {app: notes-api}}
    spec:
      initContainers:
      - name: migrate
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        args: [migrate]
        env:
        - name: NOTES_DB_URL
          valueFrom: {secretKeyRef: {name: notes-db, key: url}}
      containers:
      - name: notes-api
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        ports: [{containerPort: 8080, name: http}]
        env:
        - name: NOTES_DB_URL
          valueFrom: {secretKeyRef: {name: notes-db, key: url}}
        - name: NOTES_API_KEY
          valueFrom: {secretKeyRef: {name: notes-api-key, key: key}}
        readinessProbe:
          httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag20}
spec:
  selector: {app: notes-api}
  ports: [{port: 80, targetPort: http, name: http}]
EOF
cat > web.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag20}
data:
  default.conf: |
    server {
      listen 8080;
      location = / { default_type text/plain; return 200 "notes web\n"; }
      location /api/ {
        proxy_set_header X-API-Key lab-key-123;
        proxy_pass http://notes-api.tag20.svc.cluster.local/;
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag20}
spec:
  replicas: 2
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /, port: http}
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag20}
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: notes, namespace: tag20}
spec:
  ingressClassName: traefik
  rules:
  - host: notes.tag20.lab
    http:
      paths:
      - path: /
        pathType: Prefix
        backend: {service: {name: web, port: {number: 80}}}
EOF
kubectl apply -f postgres.yaml -f notes-api.yaml -f web.yaml
kubectl -n tag20 rollout status statefulset/postgres --timeout=180s
kubectl -n tag20 rollout status deploy/notes-api --timeout=180s | tail -1
kubectl -n tag20 rollout status deploy/web --timeout=180s

Expected (abridged):

namespace/tag20 created
secret/notes-db created
secret/notes-api-key created
statefulset.apps/postgres created
...
ingress.networking.k8s.io/notes created
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...
deployment "notes-api" successfully rolled out
deployment "web" successfully rolled out

Create a note, then take the first measurement:

# on cp1
U='--resolve notes.tag20.lab:80:172.16.0.126 http://notes.tag20.lab'
curl -s -m 10 $U/api/notes -d '{"text":"Wettkampf"}'; echo
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Expected:

{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1

This is the healthy playing field. The manifests in ~/debug/tag20 are the desired state – comparing them with the live object is allowed and often the fastest way.

Round 1: The new version

Situation: The team has rolled out notes-api again. The pipeline has been waiting on rollout status for minutes.

Referee commands:

# on cp1
kubectl -n tag20 create quota grenze --hard=pods=5
kubectl -n tag20 rollout restart deploy/notes-api

Target time: 4 min

Hint: The new Pod is not missing because of the Pod – ask the ReplicaSet.

Solution Round 1

# on cp1
sleep 20
kubectl -n tag20 rollout status deploy/notes-api --timeout=5s
kubectl -n tag20 get rs -l app=notes-api
kubectl -n tag20 events --types=Warning | grep FailedCreate | tail -1 | cut -c1-260 \
  | fold -w 86
kubectl -n tag20 describe quota grenze | tail -3

Expected (line wrapped):

Waiting for deployment "notes-api" rollout to finish: 0 out of 2 new replicas have been
 updated...
error: timed out waiting for the condition
NAME                   DESIRED   CURRENT   READY   AGE
notes-api-6c5f9d64bc   1         0         0       25s
notes-api-6d87674cb5   2         2         2       73s
6s (x4 over 24s)    Warning   FailedCreate   ReplicaSet/notes-api-6c5f9d64bc   (combin
ed from similar events): Error creating: pods "notes-api-6c5f9d64bc-8bxpv" is forbidde
n: exceeded quota: grenze, requested: pods=1, used: pods=5, limited: pods=5
Resource    Used  Hard
--------    ----  ----
pods        5     5

The quota allows exactly as many Pods as are running. A rollout briefly needs one more Pod (maxSurge), and the quota rejects it at the ReplicaSet (Day 3). The quota stays – just with room for rollouts:

# on cp1
kubectl -n tag20 patch resourcequota grenze -p '{"spec":{"hard":{"pods":"8"}}}'
kubectl -n tag20 rollout status deploy/notes-api --timeout=120s | tail -1

Expected:

resourcequota/grenze patched
deployment "notes-api" successfully rolled out

Measurement (commands from the rules of the game):

# on cp1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Expected:

[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1

Save your progress? Sign up for free

Round 2: 503

Situation: “I only made the labels consistent.” Since then the site answers with 503 – even the start page.

Referee commands:

# on cp1
kubectl -n tag20 patch service web -p '{"spec":{"selector":{"app":"wbe"}}}'

Target time: 3 min

Hint: Who delivers the 503 – nginx or Traefik? And where does Traefik want to go?

Solution Round 2

# on cp1
sleep 10
curl -s -m 10 -w ' %{http_code}\n' $U/
kubectl -n tag20 get endpointslices -l kubernetes.io/service-name=web
kubectl -n tag20 get svc web -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n tag20 get pods -l app=web --no-headers | wc -l

Expected:

no available server
 503
NAME        ADDRESSTYPE   PORTS     ENDPOINTS   AGE
web-hbj62   IPv4          <unset>   <unset>     113s
{"app":"wbe"}
2

no available server is the answer from Traefik itself: the Service has no endpoints because its selector matches no Pod (Day 7). Back to the desired value from web.yaml:

# on cp1
kubectl -n tag20 patch service web -p '{"spec":{"selector":{"app":"web"}}}'
sleep 3
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Expected:

service/web patched
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1

Save your progress? Sign up for free

Round 3: The unified policy

Situation: The platform team has rolled out a Kyverno policy “to unify the API keys”. Afterwards, notes-api was restarted – and is stuck.

Referee commands:

# on cp1
cat > /tmp/mpol.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata: {name: api-key-vereinheitlichen}
spec:
  matchConstraints:
    namespaceSelector:
      matchLabels: {kubernetes.io/metadata.name: tag20}
    objectSelector:
      matchLabels: {app: notes-api}
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE]
      resources: [pods]
  mutations:
  - patchType: JSONPatch
    jsonPatch:
      expression: >-
        [JSONPatch{op: "replace",
          path: "/spec/containers/0/env/1/valueFrom/secretKeyRef/key",
          value: "api-key"}]
EOF
kubectl apply -f /tmp/mpol.yaml && rm /tmp/mpol.yaml
sleep 5
kubectl -n tag20 rollout restart deploy/notes-api

Target time: 5 min

Hint: Compare the Pod template in the Deployment with the Pod that is stuck.

Solution Round 3

# on cp1
sleep 30
kubectl -n tag20 get pods -l app=notes-api
kubectl -n tag20 events --types=Warning | grep -m1 "couldn't find key" \
  | grep -o 'Error:.*'
J='{.spec.template.spec.containers[0].env[1].valueFrom.secretKeyRef}{"\n"}'
kubectl -n tag20 get deploy notes-api -o jsonpath="$J"
P=$(kubectl -n tag20 get pods -l app=notes-api --no-headers | grep -v Running \
  | awk '{print $1}')
kubectl -n tag20 get pod $P \
  -o jsonpath='{.spec.containers[0].env[1].valueFrom.secretKeyRef}{"\n"}'
kubectl get mpol

Expected:

NAME                         READY   STATUS                       RESTARTS   AGE
notes-api-67f8dc996-6m9fz    0/1     CreateContainerConfigError   0          30s
notes-api-6c5f9d64bc-5p7k7   1/1     Running                      0          64s
notes-api-6c5f9d64bc-ffm7k   1/1     Running                      0          60s
Error: couldn't find key api-key in Secret tag20/notes-api-key
{"key":"key","name":"notes-api-key"}
{"key":"api-key","name":"notes-api-key"}
NAME                       AGE   READY
api-key-vereinheitlichen   37s   true

CreateContainerConfigError with a missing key (Day 4) – but the Deployment asks for the correct key key. The Pod only got the wrong one when it was created: a mutation by Kyverno (Day 12). That is why the manifest looks clean, only the live object does not. The policy is the cause; it asks for a key that the Secret does not have. Remove it (or correct it with the platform team), then have the stuck Pod recreated:

# on cp1
kubectl delete mpol api-key-vereinheitlichen
kubectl -n tag20 delete pod $P
kubectl -n tag20 rollout status deploy/notes-api --timeout=120s | tail -1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Expected:

mutatingpolicy.policies.kyverno.io "api-key-vereinheitlichen" deleted
pod "notes-api-67f8dc996-6m9fz" deleted from tag20 namespace
deployment "notes-api" successfully rolled out
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1

Save your progress? Sign up for free

Round 4: Only the database

Situation: Security has restricted notes-api “to the bare minimum”. It still works. Only after the next restart does someone call.

Referee commands:

# on cp1
cat > /tmp/np.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: nur-datenbank, namespace: tag20}
spec:
  podSelector: {matchLabels: {app: notes-api}}
  policyTypes: [Egress]
  egress:
  - to:
    - podSelector: {matchLabels: {app: postgres}}
    ports: [{port: 5432, protocol: TCP}]
EOF
kubectl apply -f /tmp/np.yaml && rm /tmp/np.yaml
kubectl -n tag20 rollout restart deploy/notes-api

Target time: 5 min

Hint: Read the log of the init container – which address does it want to reach?

Solution Round 4

# on cp1
sleep 40
kubectl -n tag20 get pods -l app=notes-api
P=$(kubectl -n tag20 get pods -l app=notes-api --no-headers | grep Init \
  | awk '{print $1}')
kubectl -n tag20 logs $P -c migrate --tail 1 | cut -c1-260 | fold -w 86
kubectl -n tag20 get networkpolicy

Expected:

NAME                         READY   STATUS       RESTARTS      AGE
notes-api-6464cd5898-gl95n   0/1     Init:Error   2 (32s ago)   40s
notes-api-67f8dc996-7vfx7    1/1     Running      0             50s
notes-api-67f8dc996-ljg6g    1/1     Running      0             54s
time=2026-11-20T14:13:35.067Z level=ERROR msg="database unreachable" source=env err="f
ailed to connect to `user=notes database=notes`: hostname resolving error: lookup post
gres on 10.43.0.10:53: dial udp 10.43.0.10:53: i/o timeout"
NAME            POD-SELECTOR    AGE
nur-datenbank   app=notes-api   42s

The database is allowed, but not the way to its name: 10.43.0.10:53 is CoreDNS (Day 8). The old Pods kept running because their connections were already established – only a restart has to resolve the name again. The policy stays, it just gets DNS added:

# on cp1
cat > np-fix.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: nur-datenbank, namespace: tag20}
spec:
  podSelector: {matchLabels: {app: notes-api}}
  policyTypes: [Egress]
  egress:
  - to:
    - podSelector: {matchLabels: {app: postgres}}
    ports: [{port: 5432, protocol: TCP}]
  - to:
    - namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: kube-system}
      podSelector: {matchLabels: {k8s-app: kube-dns}}
    ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]
EOF
kubectl apply -f np-fix.yaml
kubectl -n tag20 rollout status deploy/notes-api --timeout=180s | tail -1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Expected:

networkpolicy.networking.k8s.io/nur-datenbank configured
deployment "notes-api" successfully rolled out
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1

The init container is stuck in backoff (Day 5); it can take up to a minute until it tries again.

Save your progress? Sign up for free

Round 5: 404

Situation: “We are moving to ingress-nginx soon, I already switched the Ingress over.”

Referee commands:

# on cp1
kubectl -n tag20 patch ingress notes --type=merge \
  -p '{"spec":{"ingressClassName":"nginx"}}'

Target time: 3 min

Hint: Which IngressClasses exist in the cluster?

Solution Round 5

# on cp1
sleep 5
curl -s -m 10 -w ' %{http_code}\n' $U/api/notes
kubectl -n tag20 get ingress notes
kubectl get ingressclass

Expected:

404 page not found
 404
NAME    CLASS   HOSTS             ADDRESS   PORTS   AGE
notes   nginx   notes.tag20.lab             80      4m6s
NAME                CONTROLLER                      PARAMETERS   AGE
traefik (default)   traefik.io/ingress-controller   <none>       7h41m

404 page not found is Traefik’s answer for “no route”: it silently ignores an Ingress with a foreign class, without an event (Day 9). There is no controller for nginx:

# on cp1
kubectl -n tag20 patch ingress notes --type=merge \
  -p '{"spec":{"ingressClassName":"traefik"}}'
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Expected:

ingress.networking.k8s.io/notes patched
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1

Save your progress? Sign up for free

Round 6: The database moves

Situation: “Postgres is supposed to go to the node with the faster disk.” Since then, requests to notes hang until the timeout. The referee picks a worker on which Postgres is currently not running.

Referee commands:

# on cp1
N=$(kubectl -n tag20 get pod postgres-0 -o jsonpath='{.spec.nodeName}')
A=$(kubectl get nodes -o name | cut -d/ -f2 | grep -v -x -e cp1 -e $N | head -1)
kubectl -n tag20 patch statefulset postgres \
  -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"'$A'"}}}}}'
kubectl -n tag20 delete pod postgres-0

Target time: 5 min

Hint: Where is the volume? And does a StatefulSet replace a Pod that never starts on its own?

Solution Round 6

# on cp1
sleep 20
kubectl -n tag20 get pod postgres-0
kubectl -n tag20 events --for pod/postgres-0 | tail -1 | cut -c1-260 | fold -w 86
kubectl -n tag20 get pod postgres-0 -o jsonpath='{.spec.nodeSelector}{"\n"}'
PV=$(kubectl -n tag20 get pvc data-postgres-0 -o jsonpath='{.spec.volumeName}')
kubectl get pv $PV -o jsonpath='{.spec.nodeAffinity.required}{"\n"}' | fold -w 86

Expected:

NAME         READY   STATUS    RESTARTS   AGE
postgres-0   0/1     Pending   0          20s
20s (x2 over 20s)       Warning   FailedScheduling   Pod/postgres-0   0/3 nodes are av
ailable: 1 node(s) didn't match PersistentVolume's node affinity, 2 node(s) didn't mat
ch Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is 
no
{"kubernetes.io/hostname":"worker2"}
{"nodeSelectorTerms":[{"matchExpressions":[{"key":"kubernetes.io/hostname","operator":
"In","values":["worker1"]}]}]}

The local-path volume lives on a specific node and does not move along (Day 10); the nodeSelector points to a different one. notes-api waits for the database, nginx gives up with 504 after 60 seconds. First remove the selector – and then delete the stuck Pod: a StatefulSet does not replace a Pod with the old template as long as that Pod was never ready (the Kubernetes docs call this “forced rollback”).

# on cp1
kubectl -n tag20 patch statefulset postgres --type=json \
  -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
kubectl -n tag20 delete pod postgres-0
kubectl -n tag20 rollout status statefulset/postgres --timeout=120s | tail -1
sleep 15
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Expected:

statefulset.apps/postgres patched
pod "postgres-0" deleted from tag20 namespace
partitioned roll out complete: 1 new pods have been updated...
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1

The note is still there – the data was safe on the old node the whole time.

Save your progress? Sign up for free

Round 7: The validator

Situation: A vendor tool was installed “for testing” and removed again. Now web is supposed to get a new configuration, but no new Pod is created.

Referee commands:

# on cp1
cat > /tmp/wh.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: {name: tag20-pruefer}
webhooks:
- name: pruefer.tag20.lab
  admissionReviewVersions: [v1]
  sideEffects: None
  failurePolicy: Fail
  namespaceSelector:
    matchLabels: {kubernetes.io/metadata.name: tag20}
  rules:
  - apiGroups: [""]
    apiVersions: [v1]
    operations: [CREATE]
    resources: [pods]
  clientConfig:
    service: {name: pruefer, namespace: tag20, path: /validate}
EOF
kubectl apply -f /tmp/wh.yaml && rm /tmp/wh.yaml
kubectl -n tag20 rollout restart deploy/web

Target time: 4 min

Hint: Internal error occurred means: the rejection did not come from the API server itself.

Solution Round 7

# on cp1
sleep 15
kubectl -n tag20 get deploy web
kubectl -n tag20 events --types=Warning | grep -m1 'failed calling webhook' \
  | cut -c1-280 | fold -w 86
kubectl get validatingwebhookconfigurations -o name | grep -v kyverno
kubectl -n tag20 get svc pruefer

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    2/2     0            2           5m12s
5s (x12 over 15s)       Warning   FailedCreate       ReplicaSet/web-5489c7bf58        
 Error creating: Internal error occurred: failed calling webhook "pruefer.tag20.lab": 
failed to call webhook: Post "https://pruefer.tag20.svc:443/validate?timeout=10s": ser
vice "pruefer" not fou
validatingwebhookconfiguration.admissionregistration.k8s.io/tag20-pruefer
Error from server (NotFound): services "pruefer" not found

A webhook without a Service, failurePolicy: Fail, valid for every Pod in tag20 (Day 16). The tool is gone, its configuration is not – so it has to go as well:

# on cp1
kubectl delete validatingwebhookconfiguration tag20-pruefer
kubectl -n tag20 rollout status deploy/web --timeout=120s | tail -1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas

Expected:

validatingwebhookconfiguration.admissionregistration.k8s.io "tag20-pruefer" deleted
deployment "web" successfully rolled out
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1

Save your progress? Sign up for free

Round 8: Maintenance window

Situation: A worker is to be patched. The referee picks the worker without Postgres and starts the drain with a 40-second deadline. Then it is your turn: the node is to be emptied, notes is to stay reachable, and at the end the node is back in service.

Referee commands:

# on cp1
cat > /tmp/pdb.yaml <<'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: notes-pdb, namespace: tag20}
spec:
  maxUnavailable: 0
  selector:
    matchExpressions:
    - {key: app, operator: In, values: [web, notes-api]}
EOF
kubectl apply -f /tmp/pdb.yaml && rm /tmp/pdb.yaml
P=$(kubectl -n tag20 get pod postgres-0 -o jsonpath='{.spec.nodeName}')
W=$(kubectl get nodes -o name | cut -d/ -f2 | grep -v -x -e cp1 -e $P | head -1)
echo "Wartung: $W"
kubectl drain $W --ignore-daemonsets --delete-emptydir-data --timeout=40s 2>&1 \
  | tail -1 | cut -c1-120

Target time: 4 min

Hint: ALLOWED DISRUPTIONS.

Solution Round 8

W is the node the referee named – determined here with the same rule:

# on cp1
P=$(kubectl -n tag20 get pod postgres-0 -o jsonpath='{.spec.nodeName}')
W=$(kubectl get nodes -o name | cut -d/ -f2 | grep -v -x -e cp1 -e $P | head -1)
echo $W
kubectl get nodes
kubectl -n tag20 get pdb
kubectl -n tag20 get pods -l 'app in (web,notes-api)' -o wide | awk '{print $1, $7}'

Expected:

worker2
NAME      STATUS                     ROLES                AGE     VERSION
cp1       Ready                      control-plane,etcd   7h45m   v1.37.2+rke2r1
worker1   Ready                      <none>               7h42m   v1.37.2+rke2r1
worker2   Ready,SchedulingDisabled   <none>               7h41m   v1.37.2+rke2r1
NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
notes-pdb   N/A             0                 0                     42s
NAME NODE
notes-api-6464cd5898-gl95n worker1
notes-api-6464cd5898-sjj98 worker2
web-5489c7bf58-9gncl worker2
web-5489c7bf58-vjtlf worker1

A PDB with maxUnavailable: 0 forbids every voluntary eviction – the drain waits forever (Day 14). With two replicas, one may go:

# on cp1
kubectl -n tag20 patch pdb notes-pdb --type=merge -p '{"spec":{"maxUnavailable":1}}'
kubectl drain $W --ignore-daemonsets --delete-emptydir-data --timeout=120s \
  2>&1 | tail -1
kubectl uncordon $W
kubectl -n tag20 rollout status deploy/notes-api --timeout=120s | tail -1
kubectl -n tag20 rollout status deploy/web --timeout=120s | tail -1
sleep 5
curl -s -m 10 $U/api/notes; echo
kubectl -n tag20 get deploy,sts -o custom-columns=NAME:.metadata.name,\
READY:.status.readyReplicas,AKTUELL:.status.updatedReplicas
kubectl get nodes

Expected:

poddisruptionbudget.policy/notes-pdb patched
node/worker2 drained
node/worker2 uncordoned
deployment "notes-api" successfully rolled out
deployment "web" successfully rolled out
[{"id":"44bc0f3b-35e8-4563-b6d2-b2d36c6f921b","text":"Wettkampf"}]
NAME        READY   AKTUELL
notes-api   2       2
web         2       2
postgres    1       1
NAME      STATUS   ROLES                AGE     VERSION
cp1       Ready    control-plane,etcd   7h45m   v1.37.2+rke2r1
worker1   Ready    <none>               7h42m   v1.37.2+rke2r1
worker2   Ready    <none>               7h41m   v1.37.2+rke2r1

Save your progress? Sign up for free

Scoring

Round Fault Target time Points
1 Quota without room for maxSurge 4 min 2
2 Service selector wbe 3 min 2
3 MutatingPolicy sets the wrong Secret key 5 min 3
4 Egress policy without DNS 5 min 2
5 ingressClassName: nginx 3 min 1
6 nodeSelector against local-path volume, Pod stuck 5 min 3
7 orphaned webhook with failurePolicy: Fail 4 min 2
8 PDB maxUnavailable: 0, node not put back into service 4 min 3

Full points only within the target time, without the hint and with the measurement; otherwise half (round down). Zero points for symptom fixes: deleting the quota or PDB, removing the policy instead of adding a DNS rule, shutting down Kyverno, recreating the PVC or namespace. In round 8 you lose one point if the node is still SchedulingDisabled at the end. Total target time: 33 min, maximum: 18 points.

Points Assessment
15–18 ready for the final exam (Appendix B)
10–14 good; repeat the days of the rounds you missed
0–9 work through Days 3, 7, 8, 10, 12 and 14 again

Challenge

Bonus round without points: a colleague started a Helm chart info for notes, and the pipeline was aborted in the middle of the upgrade. Bring the release back into a state in which helm upgrade works – without touching the release Secrets.

Referee commands:

# on cp1
cd ~/debug/tag20
helm create info >/dev/null
helm install info ./info -n tag20 --wait --timeout 2m | grep STATUS
helm upgrade info ./info -n tag20 --set image.tag=gibtsnicht --wait --timeout 5m &
sleep 5
kill -9 %1

Hint: Day 17, Drill 3.

Solution

# on cp1
helm upgrade info ./info -n tag20 2>&1 | tail -1
helm -n tag20 history info -o json | jq -r '.[] | "\(.revision) \(.status)"'
helm -n tag20 rollback info 1 2>&1 | tail -1
helm upgrade info ./info -n tag20 --wait | grep -E 'STATUS|REVISION'

Expected:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
1 deployed
2 pending-upgrade
Rollback was a success! Happy Helming!
STATUS: deployed
REVISION: 4

pending-upgrade blocks every further upgrade; a rollback to the last good revision releases the lock.

Save your progress? Sign up for free

Cleanup

# on cp1
helm uninstall info -n tag20
kubectl delete namespace tag20
kubectl get pv --no-headers | wc -l

Expected:

release "info" uninstalled
namespace "tag20" deleted
No resources found
0

With the namespace, the PVC disappears, and local-path deletes the volume (reclaimPolicy: Delete).

Checkpoint

# on cp1
kubectl get nodes

Expected: three nodes Ready, none SchedulingDisabled.

# on cp1
kubectl get mpol,vpol --no-headers 2>&1; kubectl get validatingwebhookconfigurations \
  -o name | grep -c -v kyverno

Expected: No resources found and 0 – no leftovers from rounds 3 and 7.

Interview Note

  • A rollout is stuck without an error on the Pod – where do you look? At the ReplicaSet: FailedCreate caused by quota, an admission webhook or Pod Security shows up there, not on the Deployment.
  • The manifest is correct, but the Pod is still wrong? Mutating admission (Kyverno, webhooks): compare the live object with the manifest, kubectl get mpol or mutatingwebhookconfigurations.
  • Why does a StatefulSet Pod not restart after the fix? A Pod with the old template that was never ready is not replaced automatically – delete it, then the new template takes effect.

Enterprise Note

  • Play game days like this one regularly with the on-call team; the solutions become runbooks.
  • Always plan quotas with headroom for maxSurge and with an alert on FailedCreate.
  • Roll out mutating policies only with a review and a test run (kyverno test, Day 12); keep an eye on the policy reports.
  • Every tool that installs webhooks belongs in an inventory with an owner; find orphaned webhook configurations via policy or alert.
  • PDBs belong in code review: maxUnavailable: 0 or minAvailable equal to the replica count blocks every maintenance.