Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Find and fix ten real disruptions in the RKE2 cluster under time pressure: images, registry, quota, budgets, nodes, network, Secrets, Gateway, maintenance and an accidental helm uninstall. · Duration: approx. 150 min · Prerequisite: Day 14

Briefing

  • Every round starts with a Game master block: these commands build in the fault. After that you only read the Situation and work against the clock.
  • Search on your own first, then read the solution. The solution shows one way, not the only one.
  • The measurement loop from Day 14 keeps running in the second terminal. It shows whether your users notice anything.
  • Tools of choice: kubectl get, describe, logs --previous, get events, helm history, helm get values, journalctl on the node.
 read situation ─► see symptom ─► find layer ─► cause ─► fix ─► cross-check
                   (loop,          (Pod, node,
                    get pods)       Helm, network)

Start of the day as on Day 14, with the state of Day 14:

source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
mkdir -p "$LAB/tag15" && cd "$LAB/tag15"
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag11/notes/." .
cp -R "$LAB/code/tag13/notes/." . && cp -R "$LAB/code/tag14/notes/." .
sed -i '' "s/SERVER_IP/$SERVER/" values-rke2.yaml
kubectl get nodes
helm list

Expected (abridged, names, IPs and times will differ on your machine):

NAME          STATUS   ROLES                AGE   VERSION
rke2-agent1   Ready    <none>               1h    v1.37.1+rke2r1
rke2-agent2   Ready    <none>               1h    v1.37.1+rke2r1
rke2-server   Ready    control-plane,etcd   1h    v1.37.1+rke2r1
NAME    NAMESPACE   REVISION   UPDATED   STATUS     CHART         APP VERSION
notes   notes       …          2026-…    deployed   notes-0.3.0   0.2.0

Round 1: The Version That Doesn’t Exist

Game master:

helm upgrade notes chart/notes -f values-rke2.yaml --set web.image.tag=0.1.1 \
  --wait --timeout 90s

Situation: A colleague wanted to bump web to “the new version”. After a minute and a half, Helm reports an error, but the loop keeps showing 200. Find out what went wrong, and restore the old state cleanly. Time limit: 5 min.

helm history notes | tail -2
kubectl get pods -l app.kubernetes.io/name=web

Expected (abridged):

Error: UPGRADE FAILED: resource Deployment/notes/web not ready. status: InProgress,
message: Updated: 1/2
context deadline exceeded
3   Sat Oct 31 10:23:19 2026   deployed   notes-0.3.0   0.2.0   Upgrade complete
4   Sat Oct 31 11:01:15 2026   failed     notes-0.3.0   0.2.0   Upgrade "notes" fai…
NAME                   READY   STATUS             RESTARTS      AGE
web-5bcd78bdd9-n94xv   1/1     Running            3 (15m ago)   39m
web-5bcd78bdd9-ww9zl   1/1     Running            2 (8m ago)    17m
web-757b4c5488-bfhkr   0/1     ImagePullBackOff   0             91s

Solution

kubectl get events --field-selector reason=Failed -o custom-columns=MSG:.message \
  | grep -m1 0.1.1
helm rollback notes --wait
helm history notes | tail -1
kubectl get pods -l app.kubernetes.io/name=web

Expected (abridged):

Failed to pull image "…:5000/notes-web:0.1.1": rpc error: code = NotFound desc = …
…:5000/notes-web:0.1.1: not found
Rollback was a success! Happy Helming!
5   Sat Oct 31 11:02:54 2026   deployed   notes-0.3.0   0.2.0   Rollback to 3
NAME                   READY   STATUS        RESTARTS      AGE
web-5bcd78bdd9-n94xv   1/1     Running       3 (15m ago)   39m
web-5bcd78bdd9-ww9zl   1/1     Running       2 (8m ago)    17m
web-757b4c5488-bfhkr   0/1     Terminating   0             93s

maxUnavailable: 0 left the old Pods standing, which is why users noticed nothing. helm rollback without a number goes back one revision. A rollback creates a new revision; the history is preserved.

Save your progress? Sign up for free

Round 2: Only One Node Refuses

Game master:

multipass exec rke2-agent2 -- sudo sed -i 's/lab-registry-pw-123/lab-registry-pw-456/' \
  /etc/rancher/rke2/registries.yaml
multipass exec rke2-agent2 -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node --all --timeout=180s

Situation: “No new Pods start on rke2-agent2 anymore, but on the others they do.” Prove it with a test Pod, find the cause and fix it. Time limit: 10 min.

kubectl -n default run pulltest --image="$SERVER:5000/notes-api:0.2.0" \
  --image-pull-policy=Always --env NOTES_API_KEY=lab-api-key-123 \
  --overrides='{"spec":{"nodeName":"rke2-agent2"}}'
kubectl -n default describe pod pulltest | grep -m1 'Failed to pull'

Expected (abridged):

pod/pulltest created
Warning  Failed  1s  kubelet  spec.containers{pulltest}: Failed to pull image
"…:5000/notes-api:0.2.0": … unexpected status from HEAD request to
https://…:5000/v2/notes-api/manifests/0.2.0: 401 Unauthorized

Solution

401 instead of x509 (Day 13): the CA is right, the credentials aren’t. They live per node in registries.yaml:

multipass exec rke2-agent2 -- sudo grep password /etc/rancher/rke2/registries.yaml
multipass exec rke2-agent2 -- sudo sed -i 's/lab-registry-pw-456/lab-registry-pw-123/' \
  /etc/rancher/rke2/registries.yaml
multipass exec rke2-agent2 -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/rke2-agent2 --timeout=180s
kubectl -n default delete pod pulltest
kubectl -n default run pulltest --image="$SERVER:5000/notes-api:0.2.0" \
  --image-pull-policy=Always --env NOTES_API_KEY=lab-api-key-123 \
  --overrides='{"spec":{"nodeName":"rke2-agent2"}}'
kubectl -n default wait --for=condition=Ready pod/pulltest --timeout=60s
kubectl -n default delete pod pulltest

Expected (abridged):

      password: lab-registry-pw-456
node/rke2-agent2 condition met
pod "pulltest" deleted from default namespace
pod/pulltest created
pod/pulltest condition met
pod "pulltest" deleted from default namespace

nodeName bypasses the scheduler and forces the Pod onto exactly one node. For tests that is ideal; in everyday use you take nodeSelector or affinity.

Save your progress? Sign up for free

Round 3: Twenty Don’t Fit

Game master:

helm upgrade notes chart/notes -f values-rke2.yaml --set api.replicas=20 \
  --wait --timeout 60s

Situation: For a load test, the api was supposed to go to 20 replicas. Helm aborts, yet clearly more Pods are running than before. How many, why not all, and how do you get back cleanly? Time limit: 5 min.

kubectl get deploy api
kubectl get resourcequota notes

Expected (abridged):

context deadline exceeded
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    16/20   16           16          58m
NAME    REQUEST                                                          LIMIT    AGE
notes   pods: 20/20, requests.cpu: 1020m/2, requests.memory: 896Mi/2Gi   …        58m

Solution

kubectl get events --field-selector reason=FailedCreate \
  -o custom-columns=MSG:.message | grep -m1 quota
helm rollback notes --wait
kubectl get deploy api

Expected (abridged):

Error creating: pods "api-69c4fc7c84-jd4cx" is forbidden: exceeded quota: notes,
requested: pods=1, used: pods=20, limited: pods=20
Rollback was a success! Happy Helming!
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    3/3     3            3           58m

The quota from Day 6 allows 20 Pods in the namespace. The ReplicaSet doesn’t get a Pending Pod; the rejection only shows up in the events (FailedCreate). For a real load test, you raise the quota deliberately, namely in k8s/namespace.yaml.

Save your progress? Sign up for free

Round 4: The Drain That Never Ends

Game master:

helm upgrade notes chart/notes -f values-rke2.yaml --set pdb.web.minAvailable=2 \
  --wait
kubectl get pods -o wide -l 'app.kubernetes.io/name in (web,db)'

Expected (abridged):

NAME                   READY   STATUS    …   NODE
db-0                   1/1     Running   …   rke2-agent2
web-5bcd78bdd9-l2d7l   1/1     Running   …   rke2-agent1
web-5bcd78bdd9-pzbwp   1/1     Running   …   rke2-server

Situation: A node with a web Pod, but without db-0, is due for maintenance (in the book rke2-agent1). The drain hangs. Get it finished without disturbing the users. Time limit: 5 min.

kubectl drain rke2-agent1 --ignore-daemonsets --delete-emptydir-data --timeout=45s

Expected (abridged):

node/rke2-agent1 cordoned
…
evicting pod notes/web-5bcd78bdd9-l2d7l
error when evicting pods/"web-5bcd78bdd9-l2d7l" -n "notes" (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.
evicting pod notes/web-5bcd78bdd9-l2d7l
…

Solution

kubectl get pdb web
helm rollback notes --wait
kubectl get pdb web
kubectl drain rke2-agent1 --ignore-daemonsets --delete-emptydir-data | tail -1
kubectl uncordon rke2-agent1

Expected (abridged):

NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    2               N/A               0                     55m
Rollback was a success! Happy Helming!
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    1               N/A               1                     56m
node/rke2-agent1 drained
node/rke2-agent1 uncordoned

minAvailable equal to the number of replicas means ALLOWED DISRUPTIONS 0: no drain ever gets through. A budget must always release at least one Pod. Alternatively, one more replica helps.

Warning: Never choose the node with db-0 for this round. Otherwise the database goes first, the api is no longer ready, and the api budget doesn’t let anything through anymore either (Day 14, Drill 4).

Save your progress? Sign up for free

Round 5: The Agent Doesn’t Come Back

Game master:

multipass exec rke2-agent1 -- sudo sed -i 's/:9345/:9346/' /etc/rancher/rke2/config.yaml
multipass exec rke2-agent1 -- sudo systemctl restart --no-block rke2-agent

Situation: After a “small config update”, rke2-agent1 has been NotReady for a minute. Find the cause on the node. Time limit: 10 min.

kubectl get nodes
multipass exec rke2-agent1 -- systemctl is-active rke2-agent

Expected:

NAME          STATUS     ROLES                AGE   VERSION
rke2-agent1   NotReady   <none>               88m   v1.37.1+rke2r1
rke2-agent2   Ready      <none>               88m   v1.37.1+rke2r1
rke2-server   Ready      control-plane,etcd   91m   v1.37.1+rke2r1
activating

Solution

multipass exec rke2-agent1 -- sudo journalctl -u rke2-agent --since -1min \
  --no-pager | tail -2
multipass exec rke2-agent1 -- sudo cat /etc/rancher/rke2/config.yaml
multipass exec rke2-agent1 -- sudo sed -i 's/:9346/:9345/' /etc/rancher/rke2/config.yaml
multipass exec rke2-agent1 -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/rke2-agent1 --timeout=120s

Expected (abridged):

… level=error msg="Failed to validate connection to cluster at https://…:9346: failed
to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
…
server: https://…:9346
token-file: /etc/rancher/rke2/token
node/rke2-agent1 condition met

activating means: the service keeps trying endlessly. The cause only shows up in the node’s journal, not in kubectl. The address 127.0.0.1:6444 is the agent’s local proxy through which it reaches the server. The actual error is the port 9346 behind it. --no-block prevents systemctl restart from waiting for the start that never comes.

Save your progress? Sign up for free

Round 6: Radio Silence

Game master:

cat > deny-egress.yaml <<'EOF'
# deny-egress.yaml – Day 15, Round 6 (game master)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: notes
spec:
  podSelector: {}
  policyTypes: [Egress]
EOF
kubectl apply -f deny-egress.yaml
kubectl rollout restart deploy/api

Situation: A security audit “hardened outgoing traffic”. Since then the loop shows 000, and the new api Pod doesn’t start. Find the cause and restore operations. Time limit: 10 min.

kubectl get pods -l app.kubernetes.io/name=api
kubectl exec deploy/web -- wget -qO- -T 3 http://api:8080/healthz

Expected (abridged):

NAME                   READY   STATUS             RESTARTS      AGE
api-5c854d686c-5gkdr   0/1     CrashLoopBackOff   2 (21s ago)   45s
api-69c4fc7c84-9h7zv   1/1     Running            0             3m50s
api-69c4fc7c84-ldhf2   1/1     Running            0             13m
api-69c4fc7c84-rczp7   1/1     Running            0             13m
wget: bad address 'api:8080'
command terminated with exit code 1

Solution

kubectl logs api-5c854d686c-5gkdr --previous | tail -1
kubectl get networkpolicy
kubectl delete -f deny-egress.yaml
kubectl rollout status deploy/api

Expected (abridged):

… level=ERROR msg="database unreachable" … hostname resolving error: lookup db on
10.43.0.10:53: dial udp 10.43.0.10:53: i/o timeout …
NAME                   POD-SELECTOR                 AGE
api-from-clients       app.kubernetes.io/name=api   70m
db-from-api            app.kubernetes.io/name=db    70m
default-deny-egress    <none>                       60s
default-deny-ingress   <none>                       70m
web-from-gateway       app.kubernetes.io/name=web   70m
networkpolicy.networking.k8s.io "default-deny-egress" deleted from notes namespace
…
deployment "api" successfully rolled out

bad address and i/o timeout on port 53: even DNS is blocked. Egress blocks are good, but they need exceptions: at least for DNS (kube-dns in kube-system, UDP and TCP 53) and for every destination of the app. Canal, RKE2’s Pod network, enforces NetworkPolicies just like kindnet in Part 1.

Save your progress? Sign up for free

Round 7: Half Rotated

Game master:

POSTGRES_PASSWORD=lab-db-pw-456 k8s/secrets.sh
kubectl rollout restart deploy/api

Situation: The DB password was “rotated”. The loop shows 200, but the rollout of the api doesn’t make progress. Finish the rotation without bringing back the old password. Time limit: 10 min.

kubectl get pods -l app.kubernetes.io/name=api
kubectl rollout status deploy/api --timeout=5s

Expected (abridged):

secret/notes-api unchanged
secret/notes-db configured
deployment.apps/api restarted
NAME                   READY   STATUS             RESTARTS       AGE
api-5c854d686c-5gkdr   1/1     Running            4 (119s ago)   2m49s
api-5c854d686c-fbdqz   1/1     Running            0              63s
api-5c854d686c-p7h8f   1/1     Running            0              65s
api-f5fdbdc89-q9nqk    0/1     CrashLoopBackOff   3 (6s ago)     41s
error: timed out waiting for the condition

Solution

kubectl logs api-f5fdbdc89-q9nqk --previous | grep -o 'failed SASL auth[^"]*' | head -1
kubectl exec db-0 -- psql -U notes -d notes \
  -c "ALTER USER notes PASSWORD 'lab-db-pw-456'"
kubectl rollout status deploy/api
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo

Expected (abridged):

failed SASL auth: FATAL: password authentication failed for user "notes" …
ALTER ROLE
deployment "api" successfully rolled out
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

The Secret had the new password, Postgres still the old one: POSTGRES_PASSWORD only takes effect on the very first start with an empty volume. The old api Pods kept their connections and saved operations. The correct order: first the database, then the Secret, then the rollout. From here on, lab-db-pw-456 applies.

Save your progress? Sign up for free

Round 8: 404 for Everyone

Game master:

helm upgrade notes chart/notes -f values-rke2.yaml --set gateway.hostname=notes.lab \
  --wait | grep -E '^(STATUS|REVISION):'

Situation: After a “harmless” upgrade, notes only returns 404, even though all Pods are green. Time limit: 5 min.

curl -s -i --resolve "notes.localhost:30080:$SERVER" \
  http://notes.localhost:30080/api/notes | head -1
kubectl get pods

Expected (abridged):

STATUS: deployed
REVISION: 17
HTTP/1.1 404 Not Found
NAME                     READY   STATUS    RESTARTS   AGE
api-f5fdbdc89-2snjx      1/1     Running   0          2m
…

Solution

kubectl get httproute web -o jsonpath='{.spec.hostnames}'; echo
helm get values notes | grep -A1 gateway
helm rollback notes --wait
curl -s -o /dev/null -w '%{http_code}\n' --resolve "notes.localhost:30080:$SERVER" \
  http://notes.localhost:30080/api/notes

Expected:

["notes.lab"]
gateway:
  hostname: notes.lab
Rollback was a success! Happy Helming!
200

A 404 with green Pods almost always points to routing. helm get values only lists the values that someone set – the outlier stands out immediately.

Save your progress? Sign up for free

Round 9: After the Maintenance

Game master:

kubectl cordon rke2-server rke2-agent1 rke2-agent2
kubectl rollout restart deploy/web

Situation: The nightly maintenance is over. A restart of web hangs, yet the loop still shows 200. Time limit: 5 min.

kubectl get pods -l app.kubernetes.io/name=web
kubectl get nodes

Expected (abridged):

NAME                   READY   STATUS    RESTARTS   AGE
web-5b7455b6cf-dcfcl   0/1     Pending   0          16s
web-5bcd78bdd9-pzbwp   1/1     Running   0          20m
web-5bcd78bdd9-r5vk9   1/1     Running   0          6m53s
NAME          STATUS                     ROLES                AGE   VERSION
rke2-agent1   Ready,SchedulingDisabled   <none>               93m   v1.37.1+rke2r1
rke2-agent2   Ready,SchedulingDisabled   <none>               93m   v1.37.1+rke2r1
rke2-server   Ready,SchedulingDisabled   control-plane,etcd   96m   v1.37.1+rke2r1

Solution

kubectl get events --field-selector reason=FailedScheduling \
  -o custom-columns=MSG:.message | tail -1
kubectl uncordon rke2-server rke2-agent1 rke2-agent2
kubectl rollout status deploy/web

Expected (abridged):

0/3 nodes are available: 3 node(s) were unschedulable. preemption: …
node/rke2-server uncordoned
node/rke2-agent1 uncordoned
node/rke2-agent2 uncordoned
deployment "web" successfully rolled out

Every drain needs its uncordon. Running Pods notice nothing of cordon. It only becomes noticeable at the next rollout or failure, often days later.

Save your progress? Sign up for free

Round 10: Everything Gone?

Game master:

helm uninstall notes --wait

Situation: Someone typed helm uninstall in the wrong terminal. The loop shows 000. Are the notes lost? Bring notes back. Time limit: 10 min.

kubectl get pvc,secret

Expected (abridged):

release "notes" uninstalled
NAME                              STATUS   VOLUME                                     …
persistentvolumeclaim/data-db-0   Bound    pvc-f7f33bd8-7f9f-470d-92bd-39ac2b726af9   …
NAME               TYPE     DATA   AGE
secret/notes-api   Opaque   1      74m
secret/notes-db    Opaque   2      74m

Solution

helm install notes chart/notes -f values-rke2.yaml --wait --timeout 5m \
  | grep -E '^(STATUS|REVISION):'
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo

Expected:

STATUS: deployed
REVISION: 1
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

Helm doesn’t delete PVCs created from a StatefulSet’s volumeClaimTemplate, and the Secrets don’t belong to the chart at all (Day 10). That is why the new db-0 finds its volume again. The release history, however, is gone; it starts again at revision 1.

Save your progress? Sign up for free

Checkpoint

kubectl get nodes

Expected: all three Ready, none SchedulingDisabled.

NAME          STATUS   ROLES                AGE   VERSION
rke2-agent1   Ready    <none>               95m   v1.37.1+rke2r1
rke2-agent2   Ready    <none>               95m   v1.37.1+rke2r1
rke2-server   Ready    control-plane,etcd   98m   v1.37.1+rke2r1
kubectl get networkpolicy default-deny-egress

Expected: the block from Round 6 is gone.

Error from server (NotFound): … "default-deny-egress" not found
multipass exec rke2-agent2 -- sudo grep password /etc/rancher/rke2/registries.yaml

Expected: the correct pull password again.

      password: lab-registry-pw-123
helm list
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo

Expected: release notes on chart 0.3.0, and your note is there.

NAME    NAMESPACE   REVISION   UPDATED   STATUS     CHART         APP VERSION
notes   notes       1          2026-…    deployed   notes-0.3.0   0.2.0
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

Challenge

Day 14 showed: an etcd snapshot doesn’t back up notes. Back up the database with pg_dump to the Mac, delete all notes and restore the backup.

Hint: kubectl exec passes standard input and output through. With -i, a file from the Mac can also go into the Pod.

Solution

kubectl exec db-0 -- pg_dump -U notes -d notes -Fc > notes.dump
ls -l notes.dump
kubectl exec db-0 -- psql -U notes -d notes -c 'DELETE FROM notes'
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo
kubectl exec -i db-0 -- pg_restore -U notes -d notes --clean --if-exists < notes.dump
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo

Expected (abridged):

-rw-r--r--  1 …  1710 … notes.dump
DELETE 1
[]
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

-Fc is Postgres’s binary format; pg_restore --clean --if-exists drops the existing tables before restoring. Inside the Pod, psql needs no password: local connections are trusted in the official image.

Save your progress? Sign up for free

Enterprise Note

  • Runbooks: Every round of this day is a candidate for a runbook: symptom, check commands, fix, cross-check. You practice in game days, not in an emergency.
  • Observability: Instead of a measurement loop, Prometheus alerts and synthetic checks report disruptions before users call.
  • Changes: Almost every round started with a change by hand. GitOps (Argo CD, Fleet) and mandatory review for values make them visible and reversible.
  • Backups: Database backups run automatically (CronJob or operator), end up outside the cluster and are restored regularly; otherwise they are just a hope.
  • Permissions: Not everyone should be allowed to run helm uninstall in the production namespace. RBAC separates reading, rolling out and deleting.