Day 15: Competition – Disaster Drills
Goal: Find and fix ten real failures around Rancher, Fleet, catalogs, permissions, storage and access under time pressure. · Duration: approx. 150 min · Prerequisite: Day 14
Briefing
- Every round starts with a Game master block that injects the fault. After that you read only the Situation and work against the clock.
- The faults come from Days 4 to 14. If you paid attention there, you’ll recognize the patterns.
- The most important tool is the question: at which level is it stuck? Rancher, Fleet, cluster, namespace or pod.
- The measurement loop against notes on werk runs in a second terminal (Day 10, Drill 6).
Situation ─► find level ─► read status ─► cause ─► fix ─► verify
Rancher (clusters.*.cattle.io, conditions)
Fleet (GitRepo, Bundle, build job)
Cluster (nodes, agent, events)
Start of the day:
source ~/rancher-lab/setup/env.sh
mkdir -p "$LAB/tag15" && cd "$LAB/tag15"
export KUBECONFIG="$LAB/kube/rancher.yaml"
W="$LAB/kube/werk.yaml"
W_ID=$(kubectl -n fleet-default get clusters.provisioning.cattle.io werk \
-o jsonpath='{.status.clusterName}')
K_ID=$(kubectl -n fleet-default get clusters.provisioning.cattle.io k3s \
-o jsonpath='{.status.clusterName}')
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes \
-o jsonpath='{.status.display.readyBundleDeployments}'; echoExpected:
2/2
Round 1: The Wrong Branch
Game master:
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
-p '{"spec":{"branch":"master"}}'Situation: A colleague “cleaned up” the GitRepo. Since then new commits no longer arrive, even though notes is running. Time limit: 5 min.
Solution
f='{range .status.conditions[?(@.status=="False")]}'
f+='{.type}={.status} {.message}{"\n"}{end}'
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes -o jsonpath="$f"
git -C "$LAB/tag08/notes-fleet" branch
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
-p '{"spec":{"branch":"main"}}'Expected (abridged):
GitPolling=False commit not found for branch: master
…
* main
gitrepo.fleet.cattle.io/notes patched
Fleet can’t find the branch and stops polling. What has already been rolled out stays in place; that’s why you only notice the fault at the next commit.
Save your progress? Sign up for free
Round 2: The Version That Doesn’t Exist
Game master:
cd "$LAB/tag08/notes-fleet"
sed -i '' 's/ version: 0.3.1/ version: 9.9.9/' fleet.yaml
git commit -qam "Chart 9.9.9" && git push -q origin HEAD:main
cd "$LAB/tag15"Situation: After the last commit nothing happens.
The GitRepo still shows 2/2, but the change doesn’t arrive.
Find the error message. Time limit: 10 min.
Solution
kubectl -n fleet-default get jobs --sort-by=.metadata.creationTimestamp | tail -1
J=$(kubectl -n fleet-default get jobs --sort-by=.metadata.creationTimestamp -o name \
| tail -1)
kubectl -n fleet-default logs "$J" --all-containers \
| grep -o 'failed to resolve URL.*no chart version'
cd "$LAB/tag08/notes-fleet"
git revert --no-edit HEAD && git push -q origin HEAD:main
cd "$LAB/tag15"Expected (abridged):
notes-64d0d Failed 0/1 2m30s 2m30s
failed to resolve URL of repo=https://gitea.….sslip.io/api/packages/lab/helm chart=notes
version=9.9.9: no chart version
[main 877383a] Revert "Chart 9.9.9"
Fleet builds a bundle from every commit, in a job in the namespace
fleet-default. If the job fails, the old bundle stays
active, and 2/2 is misleading. The cause is in the job’s
log.
Save your progress? Sign up for free
Round 3: Switched Off by Label
Game master:
kubectl label clusters.management.cattle.io "$K_ID" notes-
kubectl -n fleet-default label clusters.provisioning.cattle.io k3s notes-Situation: “notes on k3s is gone, we didn’t deploy anything!” Bring notes back and find out whether data was lost. Time limit: 5 min.
Solution
kubectl -n fleet-default get clustergroups.fleet.cattle.io notes
kubectl --kubeconfig "$LAB/kube/k3s.yaml" -n notes get pvc
kubectl label clusters.management.cattle.io "$K_ID" notes=ja
kubectl -n fleet-default label clusters.provisioning.cattle.io k3s notes=ja
curl -s "http://notes.$K3S.sslip.io/api/notes"; echoExpected (abridged; wait half a minute before curl):
NAME CLUSTERS-READY BUNDLES-READY STATUS
notes 1/1 2/2
NAME STATUS VOLUME …
data-db-0 Bound pvc-1cc33490-4219-4c30-9a23-96c6b5e39798 …
cluster.management.cattle.io/c-m-5pjkcmvg labeled
cluster.provisioning.cattle.io/k3s labeled
[{"id":"395a46da-…","text":"läuft auf k3s"}]
The group counts only one cluster now. No label, no target; no target, no release (Day 9). The volume survived, and so did the data.
Save your progress? Sign up for free
Round 4: The Catalog Won’t Load
Game master:
PEM=$(base64 -i "$LAB/tag07/lab-ca.crt")
kubectl --kubeconfig "$W" patch clusterrepos.catalog.cattle.io lab --type merge \
-p "{\"spec\":{\"caBundle\":\"$PEM\",\"forceUpdate\":\"$(date -u +%FT%TZ)\"}}"Situation: In werk, Apps › Charts
no longer shows the repo lab. Someone “only renewed the
CA”. Time limit: 10 min.
Solution
f='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
kubectl --kubeconfig "$W" get clusterrepos.catalog.cattle.io lab -o jsonpath="$f"
DER=$(openssl x509 -in "$LAB/tag07/lab-ca.crt" -outform der | base64)
kubectl --kubeconfig "$W" patch clusterrepos.catalog.cattle.io lab --type merge \
-p "{\"spec\":{\"caBundle\":\"$DER\",\"forceUpdate\":\"$(date -u +%FT%TZ)\"}}"
kubectl --kubeconfig "$W" wait clusterrepos.catalog.cattle.io/lab \
--for=condition=Downloaded --timeout=120sExpected (abridged):
Downloaded=False x509: malformed certificate. Will retry after 5m0s
clusterrepo.catalog.cattle.io/lab patched
clusterrepo.catalog.cattle.io/lab condition met
PEM instead of DER (Day 7). The message doesn’t say which format is expected; you have to know that.
Save your progress? Sign up for free
Round 5: The Wire Is Cut
Game master:
kubectl --kubeconfig "$LAB/kube/k3s.yaml" -n cattle-system \
scale deploy/cattle-cluster-agent --replicas=0Situation: k3s shows as Unavailable in Rancher. But notes on k3s answers. What’s going on? Time limit: 5 min.
Solution
f='{range .status.conditions[?(@.type=="Ready")]}{.status} {.reason} {.message}{end}'
kubectl get clusters.management.cattle.io "$K_ID" -o jsonpath="$f"; echo
kubectl --kubeconfig "$LAB/kube/k3s.yaml" -n cattle-system \
get deploy cattle-cluster-agent
kubectl --kubeconfig "$LAB/kube/k3s.yaml" -n cattle-system \
scale deploy/cattle-cluster-agent --replicas=1Expected (abridged):
False Disconnected Cluster agent is not connected
NAME READY UP-TO-DATE AVAILABLE AGE
cattle-cluster-agent 0/0 0 0 4h
deployment.apps/cattle-cluster-agent scaled
The cattle-cluster-agent is Rancher’s only wire into the cluster (Day 4). If it’s missing, Rancher sees nothing, and the applications keep running undisturbed.
Save your progress? Sign up for free
Round 6: The Silent Deviation
Game master:
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
-p '{"spec":{"correctDrift":{"enabled":false}}}'
kubectl --kubeconfig "$W" -n notes scale deploy api --replicas=5Situation: Five api pods are running on werk, Git says two. Nobody admits to it. Show the deviation and make sure something like this gets reverted automatically in the future. Time limit: 5 min.
Solution
c='READY:.status.display.readyBundleDeployments,STATE:.status.display.state'
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes -o custom-columns="$c"
kubectl -n fleet-default get bundles.fleet.cattle.io notes \
-o jsonpath='{.status.summary.nonReadyResources[0].modifiedStatus}'; echo
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
-p '{"spec":{"correctDrift":{"enabled":true}}}'
kubectl --kubeconfig "$W" -n notes get deploy apiExpected (abridged; wait 20 seconds before the last command):
READY STATE
0/1 Modified
[{"apiVersion":"apps/v1","kind":"Deployment","name":"api","namespace":"notes",
"patch":"{\"spec\":{\"replicas\":2}}"}]
gitrepo.fleet.cattle.io/notes patched
NAME READY UP-TO-DATE AVAILABLE AGE
api 2/2 2 2 2h
Without drift correction, Fleet only reports (Day 8). Whoever
switches off correctDrift opens the door to manual
changes.
Save your progress? Sign up for free
Round 7: There’s Room, but Not for You
Game master:
cat > ns-exp.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: notes-exp
annotations:
field.cattle.io/projectId: $W_ID:p-notes
EOF
kubectl --kubeconfig "$LAB/kube/werk-anna.yaml" create -f ns-exp.yamlSituation: anna created a namespace for experiments. Every pod in it gets rejected, even though the cluster is hardly loaded. Time limit: 10 min.
Solution
kubectl --kubeconfig "$W" -n notes-exp get resourcequota
kubectl --kubeconfig "$W" get ns notes-exp \
-o jsonpath='{.metadata.annotations.cattle\.io/status}' \
| jq -r '.Conditions[] | select(.Type=="ResourceQuotaValidated") | .Message'
kubectl --kubeconfig "$W" delete ns notes-expExpected (abridged):
NAME REQUEST LIMIT AGE
default-kd8kq pods: 0/0, requests.cpu: 0/0 limits.memory: 0/0 11s
Resource quota [limits.memory=8Gi,pods=40,requests.cpu=4] exceeds project limit
namespace "notes-exp" deleted
The project has 30 pods, each namespace gets 20; a second one no longer fits (Day 6). The fix is a decision: raise the budget, lower the share or give the namespace up again.
Save your progress? Sign up for free
Round 8: The Volume That Never Comes
Game master:
cat > lp-test.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lp-test
namespace: notes
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-path
resources:
requests:
storage: 50Mi
EOF
kubectl --kubeconfig "$W" apply -f lp-test.yamlSituation: A team explicitly wants local-path
instead of Longhorn for a test. The PVC has been Pending
for minutes. Time limit: 10 min.
Solution
kubectl --kubeconfig "$W" -n notes get events \
--field-selector involvedObject.name=lp-test,reason=ProvisioningFailed \
-o jsonpath='{.items[0].message}' | grep -o 'violates PodSecurity[^,]*'
kubectl --kubeconfig "$W" -n notes delete pvc lp-testExpected (abridged):
violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container
"helper-pod" must set securityContext.allowPrivilegeEscalation=false)
persistentvolumeclaim "lp-test" deleted from notes namespace
The local-path helper pod isn’t hardened, and
local-path-storage is missing from the exemptions of
rancher-restricted (Day 14). The fix: use Longhorn or a
template of your own with an exemption for
local-path-storage.
Save your progress? Sign up for free
Round 9: Rancher Is Gone
Game master:
kubectl -n cattle-system scale deploy rancher --replicas=0Situation: The UI is gone, and a colleague urgently needs to check the notes pods on werk. Time limit: 5 min.
Solution
A="$LAB/kube/werk-ace.yaml"
kubectl --kubeconfig "$A" --context werk get nodes --request-timeout=10s
kubectl --kubeconfig "$A" --context werk-werk1 -n notes get pods | head -3
kubectl -n cattle-system scale deploy rancher --replicas=1
kubectl -n cattle-system rollout status deploy/rancher --timeout=300sExpected (abridged):
Error from server (ServiceUnavailable): the server is currently unable to handle the
request (get nodes)
NAME READY STATUS RESTARTS AGE
api-5968bfb5cc-5smz4 1/1 Running 11 (12m ago) 81m
…
deployment.apps/rancher scaled
deployment "rancher" successfully rolled out
That is exactly what the direct context from Day 14 is for. Whoever didn’t prepare it stays locked out until Rancher runs again.
Save your progress? Sign up for free
Round 10: Commits Without Effect
Game master:
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
-p '{"spec":{"paused":true}}'
cd "$LAB/tag08/notes-fleet"
sed -i '' 's/ replicas: 2/ replicas: 3/' fleet.yaml
git commit -qam "api: 3 Repliken" && git push -q origin HEAD:main
cd "$LAB/tag15"Situation: The commit with three api replicas is an hour old, two are running on werk. Git, Fleet and the cluster look healthy. Time limit: 5 min.
Solution
kubectl -n fleet-default get bundles.fleet.cattle.io notes \
-o jsonpath='{.status.display.state}'; echo
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes -o jsonpath='{.spec.paused}'
echo
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
-p '{"spec":{"paused":false}}'
cd "$LAB/tag08/notes-fleet" && git revert --no-edit HEAD && git push -q origin HEAD:main
cd "$LAB/tag15"Expected (abridged):
WaitApplied
true
gitrepo.fleet.cattle.io/notes patched
[main …] Revert "api: 3 Repliken"
The forgotten paused from Day 8: Fleet reads the commit
but doesn’t apply it (WaitApplied). Here the change is
reverted afterwards so that the state is right for Appendix B; in a real
case it would of course stay.
Save your progress? Sign up for free
Checkpoint
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes \
-o custom-columns=BRANCH:.spec.branch,PAUSED:.spec.paused\
,DRIFT:.spec.correctDrift.enabledExpected: branch main, not paused, drift correction
on.
BRANCH PAUSED DRIFT
main false true
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes \
-o jsonpath='{.status.display.readyBundleDeployments}'; echo
c='NAME:.spec.displayName,READY:.status.conditions[?(@.type=="Ready")].status'
kubectl get clusters.management.cattle.io -o custom-columns="$c"Expected: both target clusters served, all clusters reachable in Rancher.
2/2
NAME READY
k3s True
werk True
local True
curl -s "http://notes.$WERK1.sslip.io/api/notes"; echoExpected: notes on werk with the note from Day 7.
[{"id":"c0c70ff3-…","text":"aus dem eigenen Katalog"}]
Challenge
Write a short runbook “notes not responding” for your team with five check steps in a sensible order, one command each. Test it by repeating Round 5 and following only the runbook.
Hint: From the outside in: first the application, then Rancher, then Fleet, then the cluster.
Solution
cat > runbook.md <<'EOF'
# Runbook: notes not responding
1. Application: curl -s http://notes.<werk1>.sslip.io/api/notes
2. Rancher: kubectl get clusters.management.cattle.io (column Ready)
3. Fleet: kubectl -n fleet-default get gitrepos.fleet.cattle.io notes
4. Cluster: kubectl --kubeconfig kube/werk-ace.yaml --context werk-werk1 get nodes
5. Pods: kubectl --kubeconfig kube/werk-ace.yaml --context werk-werk1 \
-n notes get pods
EOF
cat runbook.mdExpected:
# Runbook: notes not responding
1. Application: curl -s http://notes.<werk1>.sslip.io/api/notes
…
5. Pods: kubectl --kubeconfig kube/werk-ace.yaml --context werk-werk1 \
-n notes get pods
Steps 4 and 5 deliberately use the direct context: a runbook must also work when Rancher itself is the failure (Round 9). In Round 5, step 2 reports the problem, all other steps are green; the agent is the fault.
Save your progress? Sign up for free
Enterprise Note
- Game days: Rounds like these, practiced regularly and announced within the team, make on-call duty calmer than any documentation.
- Alerts instead of chance: For most rounds there is
a metric: cluster
Ready, GitRepo state, bundle state, catalogDownloaded. Rancher monitoring can watch them. - Traceable changes: Branch, labels, quotas, drift
settings belong in Git with review, not in spontaneous
kubectl patchcommands. - Emergency access: Direct contexts (ACE) and a break-glass account with a sealed password are prepared before Rancher goes down.
- After every incident: A short blameless report on what happened and what will prevent it from happening again.