Day 15: Competition – Disaster Drills
Goal: Find and fix ten real disruptions in the RKE2 cluster under time pressure: images, registry, quota, budgets, nodes, network, Secrets, Gateway, maintenance and an accidental
helm uninstall. · Duration: approx. 150 min · Prerequisite: Day 14
Briefing
- Every round starts with a Game master block: these commands build in the fault. After that you only read the Situation and work against the clock.
- Search on your own first, then read the solution. The solution shows one way, not the only one.
- The measurement loop from Day 14 keeps running in the second terminal. It shows whether your users notice anything.
- Tools of choice:
kubectl get,describe,logs --previous,get events,helm history,helm get values,journalctlon the node.
read situation ─► see symptom ─► find layer ─► cause ─► fix ─► cross-check
(loop, (Pod, node,
get pods) Helm, network)
Start of the day as on Day 14, with the state of Day 14:
source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
mkdir -p "$LAB/tag15" && cd "$LAB/tag15"
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag11/notes/." .
cp -R "$LAB/code/tag13/notes/." . && cp -R "$LAB/code/tag14/notes/." .
sed -i '' "s/SERVER_IP/$SERVER/" values-rke2.yaml
kubectl get nodes
helm listExpected (abridged, names, IPs and times will differ on your machine):
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready <none> 1h v1.37.1+rke2r1
rke2-agent2 Ready <none> 1h v1.37.1+rke2r1
rke2-server Ready control-plane,etcd 1h v1.37.1+rke2r1
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
notes notes … 2026-… deployed notes-0.3.0 0.2.0
Round 1: The Version That Doesn’t Exist
Game master:
helm upgrade notes chart/notes -f values-rke2.yaml --set web.image.tag=0.1.1 \
--wait --timeout 90sSituation: A colleague wanted to bump web to “the
new version”. After a minute and a half, Helm reports an error, but the
loop keeps showing 200. Find out what went wrong, and
restore the old state cleanly. Time limit: 5 min.
helm history notes | tail -2
kubectl get pods -l app.kubernetes.io/name=webExpected (abridged):
Error: UPGRADE FAILED: resource Deployment/notes/web not ready. status: InProgress,
message: Updated: 1/2
context deadline exceeded
3 Sat Oct 31 10:23:19 2026 deployed notes-0.3.0 0.2.0 Upgrade complete
4 Sat Oct 31 11:01:15 2026 failed notes-0.3.0 0.2.0 Upgrade "notes" fai…
NAME READY STATUS RESTARTS AGE
web-5bcd78bdd9-n94xv 1/1 Running 3 (15m ago) 39m
web-5bcd78bdd9-ww9zl 1/1 Running 2 (8m ago) 17m
web-757b4c5488-bfhkr 0/1 ImagePullBackOff 0 91s
Solution
kubectl get events --field-selector reason=Failed -o custom-columns=MSG:.message \
| grep -m1 0.1.1
helm rollback notes --wait
helm history notes | tail -1
kubectl get pods -l app.kubernetes.io/name=webExpected (abridged):
Failed to pull image "…:5000/notes-web:0.1.1": rpc error: code = NotFound desc = …
…:5000/notes-web:0.1.1: not found
Rollback was a success! Happy Helming!
5 Sat Oct 31 11:02:54 2026 deployed notes-0.3.0 0.2.0 Rollback to 3
NAME READY STATUS RESTARTS AGE
web-5bcd78bdd9-n94xv 1/1 Running 3 (15m ago) 39m
web-5bcd78bdd9-ww9zl 1/1 Running 2 (8m ago) 17m
web-757b4c5488-bfhkr 0/1 Terminating 0 93s
maxUnavailable: 0 left the old Pods standing, which is
why users noticed nothing. helm rollback without a number
goes back one revision. A rollback creates a new revision; the history
is preserved.
Save your progress? Sign up for free
Round 2: Only One Node Refuses
Game master:
multipass exec rke2-agent2 -- sudo sed -i 's/lab-registry-pw-123/lab-registry-pw-456/' \
/etc/rancher/rke2/registries.yaml
multipass exec rke2-agent2 -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node --all --timeout=180sSituation: “No new Pods start on rke2-agent2 anymore, but on the others they do.” Prove it with a test Pod, find the cause and fix it. Time limit: 10 min.
kubectl -n default run pulltest --image="$SERVER:5000/notes-api:0.2.0" \
--image-pull-policy=Always --env NOTES_API_KEY=lab-api-key-123 \
--overrides='{"spec":{"nodeName":"rke2-agent2"}}'
kubectl -n default describe pod pulltest | grep -m1 'Failed to pull'Expected (abridged):
pod/pulltest created
Warning Failed 1s kubelet spec.containers{pulltest}: Failed to pull image
"…:5000/notes-api:0.2.0": … unexpected status from HEAD request to
https://…:5000/v2/notes-api/manifests/0.2.0: 401 Unauthorized
Solution
401 instead of x509 (Day 13): the CA is
right, the credentials aren’t. They live per node in
registries.yaml:
multipass exec rke2-agent2 -- sudo grep password /etc/rancher/rke2/registries.yaml
multipass exec rke2-agent2 -- sudo sed -i 's/lab-registry-pw-456/lab-registry-pw-123/' \
/etc/rancher/rke2/registries.yaml
multipass exec rke2-agent2 -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/rke2-agent2 --timeout=180s
kubectl -n default delete pod pulltest
kubectl -n default run pulltest --image="$SERVER:5000/notes-api:0.2.0" \
--image-pull-policy=Always --env NOTES_API_KEY=lab-api-key-123 \
--overrides='{"spec":{"nodeName":"rke2-agent2"}}'
kubectl -n default wait --for=condition=Ready pod/pulltest --timeout=60s
kubectl -n default delete pod pulltestExpected (abridged):
password: lab-registry-pw-456
node/rke2-agent2 condition met
pod "pulltest" deleted from default namespace
pod/pulltest created
pod/pulltest condition met
pod "pulltest" deleted from default namespace
nodeName bypasses the scheduler and forces the Pod onto
exactly one node. For tests that is ideal; in everyday use you take
nodeSelector or affinity.
Save your progress? Sign up for free
Round 3: Twenty Don’t Fit
Game master:
helm upgrade notes chart/notes -f values-rke2.yaml --set api.replicas=20 \
--wait --timeout 60sSituation: For a load test, the api was supposed to go to 20 replicas. Helm aborts, yet clearly more Pods are running than before. How many, why not all, and how do you get back cleanly? Time limit: 5 min.
kubectl get deploy api
kubectl get resourcequota notesExpected (abridged):
context deadline exceeded
NAME READY UP-TO-DATE AVAILABLE AGE
api 16/20 16 16 58m
NAME REQUEST LIMIT AGE
notes pods: 20/20, requests.cpu: 1020m/2, requests.memory: 896Mi/2Gi … 58m
Solution
kubectl get events --field-selector reason=FailedCreate \
-o custom-columns=MSG:.message | grep -m1 quota
helm rollback notes --wait
kubectl get deploy apiExpected (abridged):
Error creating: pods "api-69c4fc7c84-jd4cx" is forbidden: exceeded quota: notes,
requested: pods=1, used: pods=20, limited: pods=20
Rollback was a success! Happy Helming!
NAME READY UP-TO-DATE AVAILABLE AGE
api 3/3 3 3 58m
The quota from Day 6 allows 20 Pods in the namespace. The ReplicaSet
doesn’t get a Pending Pod; the rejection only shows up in the events
(FailedCreate). For a real load test, you raise the quota
deliberately, namely in k8s/namespace.yaml.
Save your progress? Sign up for free
Round 4: The Drain That Never Ends
Game master:
helm upgrade notes chart/notes -f values-rke2.yaml --set pdb.web.minAvailable=2 \
--wait
kubectl get pods -o wide -l 'app.kubernetes.io/name in (web,db)'Expected (abridged):
NAME READY STATUS … NODE
db-0 1/1 Running … rke2-agent2
web-5bcd78bdd9-l2d7l 1/1 Running … rke2-agent1
web-5bcd78bdd9-pzbwp 1/1 Running … rke2-server
Situation: A node with a web Pod, but without
db-0, is due for maintenance (in the book
rke2-agent1). The drain hangs. Get it finished without
disturbing the users. Time limit: 5 min.
kubectl drain rke2-agent1 --ignore-daemonsets --delete-emptydir-data --timeout=45sExpected (abridged):
node/rke2-agent1 cordoned
…
evicting pod notes/web-5bcd78bdd9-l2d7l
error when evicting pods/"web-5bcd78bdd9-l2d7l" -n "notes" (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.
evicting pod notes/web-5bcd78bdd9-l2d7l
…
Solution
kubectl get pdb web
helm rollback notes --wait
kubectl get pdb web
kubectl drain rke2-agent1 --ignore-daemonsets --delete-emptydir-data | tail -1
kubectl uncordon rke2-agent1Expected (abridged):
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
web 2 N/A 0 55m
Rollback was a success! Happy Helming!
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
web 1 N/A 1 56m
node/rke2-agent1 drained
node/rke2-agent1 uncordoned
minAvailable equal to the number of replicas means
ALLOWED DISRUPTIONS 0: no drain ever gets through. A budget
must always release at least one Pod. Alternatively, one more replica
helps.
Warning: Never choose the node with
db-0for this round. Otherwise the database goes first, the api is no longer ready, and the api budget doesn’t let anything through anymore either (Day 14, Drill 4).
Save your progress? Sign up for free
Round 5: The Agent Doesn’t Come Back
Game master:
multipass exec rke2-agent1 -- sudo sed -i 's/:9345/:9346/' /etc/rancher/rke2/config.yaml
multipass exec rke2-agent1 -- sudo systemctl restart --no-block rke2-agentSituation: After a “small config update”,
rke2-agent1 has been NotReady for a minute.
Find the cause on the node. Time limit: 10 min.
kubectl get nodes
multipass exec rke2-agent1 -- systemctl is-active rke2-agentExpected:
NAME STATUS ROLES AGE VERSION
rke2-agent1 NotReady <none> 88m v1.37.1+rke2r1
rke2-agent2 Ready <none> 88m v1.37.1+rke2r1
rke2-server Ready control-plane,etcd 91m v1.37.1+rke2r1
activating
Solution
multipass exec rke2-agent1 -- sudo journalctl -u rke2-agent --since -1min \
--no-pager | tail -2
multipass exec rke2-agent1 -- sudo cat /etc/rancher/rke2/config.yaml
multipass exec rke2-agent1 -- sudo sed -i 's/:9346/:9345/' /etc/rancher/rke2/config.yaml
multipass exec rke2-agent1 -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/rke2-agent1 --timeout=120sExpected (abridged):
… level=error msg="Failed to validate connection to cluster at https://…:9346: failed
to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
…
server: https://…:9346
token-file: /etc/rancher/rke2/token
node/rke2-agent1 condition met
activating means: the service keeps trying endlessly.
The cause only shows up in the node’s journal, not in
kubectl. The address 127.0.0.1:6444 is the
agent’s local proxy through which it reaches the server. The actual
error is the port 9346 behind it. --no-block prevents
systemctl restart from waiting for the start that never
comes.
Save your progress? Sign up for free
Round 6: Radio Silence
Game master:
cat > deny-egress.yaml <<'EOF'
# deny-egress.yaml – Day 15, Round 6 (game master)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: notes
spec:
podSelector: {}
policyTypes: [Egress]
EOF
kubectl apply -f deny-egress.yaml
kubectl rollout restart deploy/apiSituation: A security audit “hardened outgoing
traffic”. Since then the loop shows 000, and the new api
Pod doesn’t start. Find the cause and restore operations. Time
limit: 10 min.
kubectl get pods -l app.kubernetes.io/name=api
kubectl exec deploy/web -- wget -qO- -T 3 http://api:8080/healthzExpected (abridged):
NAME READY STATUS RESTARTS AGE
api-5c854d686c-5gkdr 0/1 CrashLoopBackOff 2 (21s ago) 45s
api-69c4fc7c84-9h7zv 1/1 Running 0 3m50s
api-69c4fc7c84-ldhf2 1/1 Running 0 13m
api-69c4fc7c84-rczp7 1/1 Running 0 13m
wget: bad address 'api:8080'
command terminated with exit code 1
Solution
kubectl logs api-5c854d686c-5gkdr --previous | tail -1
kubectl get networkpolicy
kubectl delete -f deny-egress.yaml
kubectl rollout status deploy/apiExpected (abridged):
… level=ERROR msg="database unreachable" … hostname resolving error: lookup db on
10.43.0.10:53: dial udp 10.43.0.10:53: i/o timeout …
NAME POD-SELECTOR AGE
api-from-clients app.kubernetes.io/name=api 70m
db-from-api app.kubernetes.io/name=db 70m
default-deny-egress <none> 60s
default-deny-ingress <none> 70m
web-from-gateway app.kubernetes.io/name=web 70m
networkpolicy.networking.k8s.io "default-deny-egress" deleted from notes namespace
…
deployment "api" successfully rolled out
bad address and i/o timeout on port 53:
even DNS is blocked. Egress blocks are good, but they need exceptions:
at least for DNS (kube-dns in kube-system, UDP and TCP 53)
and for every destination of the app. Canal, RKE2’s Pod network,
enforces NetworkPolicies just like kindnet in Part 1.
Save your progress? Sign up for free
Round 7: Half Rotated
Game master:
POSTGRES_PASSWORD=lab-db-pw-456 k8s/secrets.sh
kubectl rollout restart deploy/apiSituation: The DB password was “rotated”. The loop
shows 200, but the rollout of the api doesn’t make
progress. Finish the rotation without bringing back the old password.
Time limit: 10 min.
kubectl get pods -l app.kubernetes.io/name=api
kubectl rollout status deploy/api --timeout=5sExpected (abridged):
secret/notes-api unchanged
secret/notes-db configured
deployment.apps/api restarted
NAME READY STATUS RESTARTS AGE
api-5c854d686c-5gkdr 1/1 Running 4 (119s ago) 2m49s
api-5c854d686c-fbdqz 1/1 Running 0 63s
api-5c854d686c-p7h8f 1/1 Running 0 65s
api-f5fdbdc89-q9nqk 0/1 CrashLoopBackOff 3 (6s ago) 41s
error: timed out waiting for the condition
Solution
kubectl logs api-f5fdbdc89-q9nqk --previous | grep -o 'failed SASL auth[^"]*' | head -1
kubectl exec db-0 -- psql -U notes -d notes \
-c "ALTER USER notes PASSWORD 'lab-db-pw-456'"
kubectl rollout status deploy/api
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echoExpected (abridged):
failed SASL auth: FATAL: password authentication failed for user "notes" …
ALTER ROLE
deployment "api" successfully rolled out
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
The Secret had the new password, Postgres still the old one:
POSTGRES_PASSWORD only takes effect on the very first start
with an empty volume. The old api Pods kept their connections and saved
operations. The correct order: first the database, then the Secret, then
the rollout. From here on, lab-db-pw-456 applies.
Save your progress? Sign up for free
Round 8: 404 for Everyone
Game master:
helm upgrade notes chart/notes -f values-rke2.yaml --set gateway.hostname=notes.lab \
--wait | grep -E '^(STATUS|REVISION):'Situation: After a “harmless” upgrade, notes only returns 404, even though all Pods are green. Time limit: 5 min.
curl -s -i --resolve "notes.localhost:30080:$SERVER" \
http://notes.localhost:30080/api/notes | head -1
kubectl get podsExpected (abridged):
STATUS: deployed
REVISION: 17
HTTP/1.1 404 Not Found
NAME READY STATUS RESTARTS AGE
api-f5fdbdc89-2snjx 1/1 Running 0 2m
…
Solution
kubectl get httproute web -o jsonpath='{.spec.hostnames}'; echo
helm get values notes | grep -A1 gateway
helm rollback notes --wait
curl -s -o /dev/null -w '%{http_code}\n' --resolve "notes.localhost:30080:$SERVER" \
http://notes.localhost:30080/api/notesExpected:
["notes.lab"]
gateway:
hostname: notes.lab
Rollback was a success! Happy Helming!
200
A 404 with green Pods almost always points to routing.
helm get values only lists the values that someone set –
the outlier stands out immediately.
Save your progress? Sign up for free
Round 9: After the Maintenance
Game master:
kubectl cordon rke2-server rke2-agent1 rke2-agent2
kubectl rollout restart deploy/webSituation: The nightly maintenance is over. A
restart of web hangs, yet the loop still shows 200.
Time limit: 5 min.
kubectl get pods -l app.kubernetes.io/name=web
kubectl get nodesExpected (abridged):
NAME READY STATUS RESTARTS AGE
web-5b7455b6cf-dcfcl 0/1 Pending 0 16s
web-5bcd78bdd9-pzbwp 1/1 Running 0 20m
web-5bcd78bdd9-r5vk9 1/1 Running 0 6m53s
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready,SchedulingDisabled <none> 93m v1.37.1+rke2r1
rke2-agent2 Ready,SchedulingDisabled <none> 93m v1.37.1+rke2r1
rke2-server Ready,SchedulingDisabled control-plane,etcd 96m v1.37.1+rke2r1
Solution
kubectl get events --field-selector reason=FailedScheduling \
-o custom-columns=MSG:.message | tail -1
kubectl uncordon rke2-server rke2-agent1 rke2-agent2
kubectl rollout status deploy/webExpected (abridged):
0/3 nodes are available: 3 node(s) were unschedulable. preemption: …
node/rke2-server uncordoned
node/rke2-agent1 uncordoned
node/rke2-agent2 uncordoned
deployment "web" successfully rolled out
Every drain needs its uncordon. Running
Pods notice nothing of cordon. It only becomes noticeable
at the next rollout or failure, often days later.
Save your progress? Sign up for free
Round 10: Everything Gone?
Game master:
helm uninstall notes --waitSituation: Someone typed helm uninstall
in the wrong terminal. The loop shows 000. Are the notes
lost? Bring notes back. Time limit: 10 min.
kubectl get pvc,secretExpected (abridged):
release "notes" uninstalled
NAME STATUS VOLUME …
persistentvolumeclaim/data-db-0 Bound pvc-f7f33bd8-7f9f-470d-92bd-39ac2b726af9 …
NAME TYPE DATA AGE
secret/notes-api Opaque 1 74m
secret/notes-db Opaque 2 74m
Solution
helm install notes chart/notes -f values-rke2.yaml --wait --timeout 5m \
| grep -E '^(STATUS|REVISION):'
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echoExpected:
STATUS: deployed
REVISION: 1
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
Helm doesn’t delete PVCs created from a StatefulSet’s
volumeClaimTemplate, and the Secrets don’t belong to the
chart at all (Day 10). That is why the new db-0 finds its
volume again. The release history, however, is gone; it starts again at
revision 1.
Save your progress? Sign up for free
Checkpoint
kubectl get nodesExpected: all three Ready, none
SchedulingDisabled.
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready <none> 95m v1.37.1+rke2r1
rke2-agent2 Ready <none> 95m v1.37.1+rke2r1
rke2-server Ready control-plane,etcd 98m v1.37.1+rke2r1
kubectl get networkpolicy default-deny-egressExpected: the block from Round 6 is gone.
Error from server (NotFound): … "default-deny-egress" not found
multipass exec rke2-agent2 -- sudo grep password /etc/rancher/rke2/registries.yamlExpected: the correct pull password again.
password: lab-registry-pw-123
helm list
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echoExpected: release notes on chart 0.3.0, and your note is
there.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
notes notes 1 2026-… deployed notes-0.3.0 0.2.0
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
Challenge
Day 14 showed: an etcd snapshot doesn’t back up notes. Back up the
database with pg_dump to the Mac, delete all notes and
restore the backup.
Hint: kubectl exec passes standard
input and output through. With -i, a file from the Mac can
also go into the Pod.
Solution
kubectl exec db-0 -- pg_dump -U notes -d notes -Fc > notes.dump
ls -l notes.dump
kubectl exec db-0 -- psql -U notes -d notes -c 'DELETE FROM notes'
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo
kubectl exec -i db-0 -- pg_restore -U notes -d notes --clean --if-exists < notes.dump
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echoExpected (abridged):
-rw-r--r-- 1 … 1710 … notes.dump
DELETE 1
[]
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
-Fc is Postgres’s binary format;
pg_restore --clean --if-exists drops the existing tables
before restoring. Inside the Pod, psql needs no password:
local connections are trusted in the official image.
Save your progress? Sign up for free
Enterprise Note
- Runbooks: Every round of this day is a candidate for a runbook: symptom, check commands, fix, cross-check. You practice in game days, not in an emergency.
- Observability: Instead of a measurement loop, Prometheus alerts and synthetic checks report disruptions before users call.
- Changes: Almost every round started with a change by hand. GitOps (Argo CD, Fleet) and mandatory review for values make them visible and reversible.
- Backups: Database backups run automatically (CronJob or operator), end up outside the cluster and are restored regularly; otherwise they are just a hope.
- Permissions: Not everyone should be allowed to run
helm uninstallin the production namespace. RBAC separates reading, rolling out and deleting.