Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Find and fix ten real failures around Rancher, Fleet, catalogs, permissions, storage and access under time pressure. · Duration: approx. 150 min · Prerequisite: Day 14

Briefing

  • Every round starts with a Game master block that injects the fault. After that you read only the Situation and work against the clock.
  • The faults come from Days 4 to 14. If you paid attention there, you’ll recognize the patterns.
  • The most important tool is the question: at which level is it stuck? Rancher, Fleet, cluster, namespace or pod.
  • The measurement loop against notes on werk runs in a second terminal (Day 10, Drill 6).
 Situation ─► find level ─► read status ─► cause ─► fix ─► verify
         Rancher  (clusters.*.cattle.io, conditions)
         Fleet    (GitRepo, Bundle, build job)
         Cluster  (nodes, agent, events)

Start of the day:

source ~/rancher-lab/setup/env.sh
mkdir -p "$LAB/tag15" && cd "$LAB/tag15"
export KUBECONFIG="$LAB/kube/rancher.yaml"
W="$LAB/kube/werk.yaml"
W_ID=$(kubectl -n fleet-default get clusters.provisioning.cattle.io werk \
  -o jsonpath='{.status.clusterName}')
K_ID=$(kubectl -n fleet-default get clusters.provisioning.cattle.io k3s \
  -o jsonpath='{.status.clusterName}')
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes \
  -o jsonpath='{.status.display.readyBundleDeployments}'; echo

Expected:

2/2

Round 1: The Wrong Branch

Game master:

kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
  -p '{"spec":{"branch":"master"}}'

Situation: A colleague “cleaned up” the GitRepo. Since then new commits no longer arrive, even though notes is running. Time limit: 5 min.

Solution

f='{range .status.conditions[?(@.status=="False")]}'
f+='{.type}={.status} {.message}{"\n"}{end}'
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes -o jsonpath="$f"
git -C "$LAB/tag08/notes-fleet" branch
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
  -p '{"spec":{"branch":"main"}}'

Expected (abridged):

GitPolling=False commit not found for branch: master
…
* main
gitrepo.fleet.cattle.io/notes patched

Fleet can’t find the branch and stops polling. What has already been rolled out stays in place; that’s why you only notice the fault at the next commit.

Save your progress? Sign up for free

Round 2: The Version That Doesn’t Exist

Game master:

cd "$LAB/tag08/notes-fleet"
sed -i '' 's/  version: 0.3.1/  version: 9.9.9/' fleet.yaml
git commit -qam "Chart 9.9.9" && git push -q origin HEAD:main
cd "$LAB/tag15"

Situation: After the last commit nothing happens. The GitRepo still shows 2/2, but the change doesn’t arrive. Find the error message. Time limit: 10 min.

Solution

kubectl -n fleet-default get jobs --sort-by=.metadata.creationTimestamp | tail -1
J=$(kubectl -n fleet-default get jobs --sort-by=.metadata.creationTimestamp -o name \
  | tail -1)
kubectl -n fleet-default logs "$J" --all-containers \
  | grep -o 'failed to resolve URL.*no chart version'
cd "$LAB/tag08/notes-fleet"
git revert --no-edit HEAD && git push -q origin HEAD:main
cd "$LAB/tag15"

Expected (abridged):

notes-64d0d   Failed   0/1           2m30s      2m30s
failed to resolve URL of repo=https://gitea.….sslip.io/api/packages/lab/helm chart=notes
version=9.9.9: no chart version
[main 877383a] Revert "Chart 9.9.9"

Fleet builds a bundle from every commit, in a job in the namespace fleet-default. If the job fails, the old bundle stays active, and 2/2 is misleading. The cause is in the job’s log.

Save your progress? Sign up for free

Round 3: Switched Off by Label

Game master:

kubectl label clusters.management.cattle.io "$K_ID" notes-
kubectl -n fleet-default label clusters.provisioning.cattle.io k3s notes-

Situation: “notes on k3s is gone, we didn’t deploy anything!” Bring notes back and find out whether data was lost. Time limit: 5 min.

Solution

kubectl -n fleet-default get clustergroups.fleet.cattle.io notes
kubectl --kubeconfig "$LAB/kube/k3s.yaml" -n notes get pvc
kubectl label clusters.management.cattle.io "$K_ID" notes=ja
kubectl -n fleet-default label clusters.provisioning.cattle.io k3s notes=ja
curl -s "http://notes.$K3S.sslip.io/api/notes"; echo

Expected (abridged; wait half a minute before curl):

NAME    CLUSTERS-READY   BUNDLES-READY   STATUS
notes   1/1              2/2
NAME        STATUS   VOLUME                                     …
data-db-0   Bound    pvc-1cc33490-4219-4c30-9a23-96c6b5e39798   …
cluster.management.cattle.io/c-m-5pjkcmvg labeled
cluster.provisioning.cattle.io/k3s labeled
[{"id":"395a46da-…","text":"läuft auf k3s"}]

The group counts only one cluster now. No label, no target; no target, no release (Day 9). The volume survived, and so did the data.

Save your progress? Sign up for free

Round 4: The Catalog Won’t Load

Game master:

PEM=$(base64 -i "$LAB/tag07/lab-ca.crt")
kubectl --kubeconfig "$W" patch clusterrepos.catalog.cattle.io lab --type merge \
  -p "{\"spec\":{\"caBundle\":\"$PEM\",\"forceUpdate\":\"$(date -u +%FT%TZ)\"}}"

Situation: In werk, Apps › Charts no longer shows the repo lab. Someone “only renewed the CA”. Time limit: 10 min.

Solution

f='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
kubectl --kubeconfig "$W" get clusterrepos.catalog.cattle.io lab -o jsonpath="$f"
DER=$(openssl x509 -in "$LAB/tag07/lab-ca.crt" -outform der | base64)
kubectl --kubeconfig "$W" patch clusterrepos.catalog.cattle.io lab --type merge \
  -p "{\"spec\":{\"caBundle\":\"$DER\",\"forceUpdate\":\"$(date -u +%FT%TZ)\"}}"
kubectl --kubeconfig "$W" wait clusterrepos.catalog.cattle.io/lab \
  --for=condition=Downloaded --timeout=120s

Expected (abridged):

Downloaded=False x509: malformed certificate. Will retry after 5m0s
clusterrepo.catalog.cattle.io/lab patched
clusterrepo.catalog.cattle.io/lab condition met

PEM instead of DER (Day 7). The message doesn’t say which format is expected; you have to know that.

Save your progress? Sign up for free

Round 5: The Wire Is Cut

Game master:

kubectl --kubeconfig "$LAB/kube/k3s.yaml" -n cattle-system \
  scale deploy/cattle-cluster-agent --replicas=0

Situation: k3s shows as Unavailable in Rancher. But notes on k3s answers. What’s going on? Time limit: 5 min.

Solution

f='{range .status.conditions[?(@.type=="Ready")]}{.status} {.reason} {.message}{end}'
kubectl get clusters.management.cattle.io "$K_ID" -o jsonpath="$f"; echo
kubectl --kubeconfig "$LAB/kube/k3s.yaml" -n cattle-system \
  get deploy cattle-cluster-agent
kubectl --kubeconfig "$LAB/kube/k3s.yaml" -n cattle-system \
  scale deploy/cattle-cluster-agent --replicas=1

Expected (abridged):

False Disconnected Cluster agent is not connected
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
cattle-cluster-agent   0/0     0            0           4h
deployment.apps/cattle-cluster-agent scaled

The cattle-cluster-agent is Rancher’s only wire into the cluster (Day 4). If it’s missing, Rancher sees nothing, and the applications keep running undisturbed.

Save your progress? Sign up for free

Round 6: The Silent Deviation

Game master:

kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
  -p '{"spec":{"correctDrift":{"enabled":false}}}'
kubectl --kubeconfig "$W" -n notes scale deploy api --replicas=5

Situation: Five api pods are running on werk, Git says two. Nobody admits to it. Show the deviation and make sure something like this gets reverted automatically in the future. Time limit: 5 min.

Solution

c='READY:.status.display.readyBundleDeployments,STATE:.status.display.state'
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes -o custom-columns="$c"
kubectl -n fleet-default get bundles.fleet.cattle.io notes \
  -o jsonpath='{.status.summary.nonReadyResources[0].modifiedStatus}'; echo
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
  -p '{"spec":{"correctDrift":{"enabled":true}}}'
kubectl --kubeconfig "$W" -n notes get deploy api

Expected (abridged; wait 20 seconds before the last command):

READY   STATE
0/1     Modified
[{"apiVersion":"apps/v1","kind":"Deployment","name":"api","namespace":"notes",
"patch":"{\"spec\":{\"replicas\":2}}"}]
gitrepo.fleet.cattle.io/notes patched
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    2/2     2            2           2h

Without drift correction, Fleet only reports (Day 8). Whoever switches off correctDrift opens the door to manual changes.

Save your progress? Sign up for free

Round 7: There’s Room, but Not for You

Game master:

cat > ns-exp.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: notes-exp
  annotations:
    field.cattle.io/projectId: $W_ID:p-notes
EOF
kubectl --kubeconfig "$LAB/kube/werk-anna.yaml" create -f ns-exp.yaml

Situation: anna created a namespace for experiments. Every pod in it gets rejected, even though the cluster is hardly loaded. Time limit: 10 min.

Solution

kubectl --kubeconfig "$W" -n notes-exp get resourcequota
kubectl --kubeconfig "$W" get ns notes-exp \
  -o jsonpath='{.metadata.annotations.cattle\.io/status}' \
  | jq -r '.Conditions[] | select(.Type=="ResourceQuotaValidated") | .Message'
kubectl --kubeconfig "$W" delete ns notes-exp

Expected (abridged):

NAME            REQUEST                        LIMIT                AGE
default-kd8kq   pods: 0/0, requests.cpu: 0/0   limits.memory: 0/0   11s
Resource quota [limits.memory=8Gi,pods=40,requests.cpu=4] exceeds project limit
namespace "notes-exp" deleted

The project has 30 pods, each namespace gets 20; a second one no longer fits (Day 6). The fix is a decision: raise the budget, lower the share or give the namespace up again.

Save your progress? Sign up for free

Round 8: The Volume That Never Comes

Game master:

cat > lp-test.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lp-test
  namespace: notes
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: local-path
  resources:
    requests:
      storage: 50Mi
EOF
kubectl --kubeconfig "$W" apply -f lp-test.yaml

Situation: A team explicitly wants local-path instead of Longhorn for a test. The PVC has been Pending for minutes. Time limit: 10 min.

Solution

kubectl --kubeconfig "$W" -n notes get events \
  --field-selector involvedObject.name=lp-test,reason=ProvisioningFailed \
  -o jsonpath='{.items[0].message}' | grep -o 'violates PodSecurity[^,]*'
kubectl --kubeconfig "$W" -n notes delete pvc lp-test

Expected (abridged):

violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container
"helper-pod" must set securityContext.allowPrivilegeEscalation=false)
persistentvolumeclaim "lp-test" deleted from notes namespace

The local-path helper pod isn’t hardened, and local-path-storage is missing from the exemptions of rancher-restricted (Day 14). The fix: use Longhorn or a template of your own with an exemption for local-path-storage.

Save your progress? Sign up for free

Round 9: Rancher Is Gone

Game master:

kubectl -n cattle-system scale deploy rancher --replicas=0

Situation: The UI is gone, and a colleague urgently needs to check the notes pods on werk. Time limit: 5 min.

Solution

A="$LAB/kube/werk-ace.yaml"
kubectl --kubeconfig "$A" --context werk get nodes --request-timeout=10s
kubectl --kubeconfig "$A" --context werk-werk1 -n notes get pods | head -3
kubectl -n cattle-system scale deploy rancher --replicas=1
kubectl -n cattle-system rollout status deploy/rancher --timeout=300s

Expected (abridged):

Error from server (ServiceUnavailable): the server is currently unable to handle the
request (get nodes)
NAME                   READY   STATUS    RESTARTS       AGE
api-5968bfb5cc-5smz4   1/1     Running   11 (12m ago)   81m
…
deployment.apps/rancher scaled
deployment "rancher" successfully rolled out

That is exactly what the direct context from Day 14 is for. Whoever didn’t prepare it stays locked out until Rancher runs again.

Save your progress? Sign up for free

Round 10: Commits Without Effect

Game master:

kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
  -p '{"spec":{"paused":true}}'
cd "$LAB/tag08/notes-fleet"
sed -i '' 's/      replicas: 2/      replicas: 3/' fleet.yaml
git commit -qam "api: 3 Repliken" && git push -q origin HEAD:main
cd "$LAB/tag15"

Situation: The commit with three api replicas is an hour old, two are running on werk. Git, Fleet and the cluster look healthy. Time limit: 5 min.

Solution

kubectl -n fleet-default get bundles.fleet.cattle.io notes \
  -o jsonpath='{.status.display.state}'; echo
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes -o jsonpath='{.spec.paused}'
echo
kubectl -n fleet-default patch gitrepos.fleet.cattle.io notes --type merge \
  -p '{"spec":{"paused":false}}'
cd "$LAB/tag08/notes-fleet" && git revert --no-edit HEAD && git push -q origin HEAD:main
cd "$LAB/tag15"

Expected (abridged):

WaitApplied
true
gitrepo.fleet.cattle.io/notes patched
[main …] Revert "api: 3 Repliken"

The forgotten paused from Day 8: Fleet reads the commit but doesn’t apply it (WaitApplied). Here the change is reverted afterwards so that the state is right for Appendix B; in a real case it would of course stay.

Save your progress? Sign up for free

Checkpoint

kubectl -n fleet-default get gitrepos.fleet.cattle.io notes \
  -o custom-columns=BRANCH:.spec.branch,PAUSED:.spec.paused\
,DRIFT:.spec.correctDrift.enabled

Expected: branch main, not paused, drift correction on.

BRANCH   PAUSED   DRIFT
main     false    true
kubectl -n fleet-default get gitrepos.fleet.cattle.io notes \
  -o jsonpath='{.status.display.readyBundleDeployments}'; echo
c='NAME:.spec.displayName,READY:.status.conditions[?(@.type=="Ready")].status'
kubectl get clusters.management.cattle.io -o custom-columns="$c"

Expected: both target clusters served, all clusters reachable in Rancher.

2/2
NAME    READY
k3s     True
werk    True
local   True
curl -s "http://notes.$WERK1.sslip.io/api/notes"; echo

Expected: notes on werk with the note from Day 7.

[{"id":"c0c70ff3-…","text":"aus dem eigenen Katalog"}]

Challenge

Write a short runbook “notes not responding” for your team with five check steps in a sensible order, one command each. Test it by repeating Round 5 and following only the runbook.

Hint: From the outside in: first the application, then Rancher, then Fleet, then the cluster.

Solution

cat > runbook.md <<'EOF'
# Runbook: notes not responding
1. Application: curl -s http://notes.<werk1>.sslip.io/api/notes
2. Rancher:   kubectl get clusters.management.cattle.io (column Ready)
3. Fleet:     kubectl -n fleet-default get gitrepos.fleet.cattle.io notes
4. Cluster:   kubectl --kubeconfig kube/werk-ace.yaml --context werk-werk1 get nodes
5. Pods:      kubectl --kubeconfig kube/werk-ace.yaml --context werk-werk1 \
              -n notes get pods
EOF
cat runbook.md

Expected:

# Runbook: notes not responding
1. Application: curl -s http://notes.<werk1>.sslip.io/api/notes
…
5. Pods:      kubectl --kubeconfig kube/werk-ace.yaml --context werk-werk1 \
              -n notes get pods

Steps 4 and 5 deliberately use the direct context: a runbook must also work when Rancher itself is the failure (Round 9). In Round 5, step 2 reports the problem, all other steps are green; the agent is the fault.

Save your progress? Sign up for free

Enterprise Note

  • Game days: Rounds like these, practiced regularly and announced within the team, make on-call duty calmer than any documentation.
  • Alerts instead of chance: For most rounds there is a metric: cluster Ready, GitRepo state, bundle state, catalog Downloaded. Rancher monitoring can watch them.
  • Traceable changes: Branch, labels, quotas, drift settings belong in Git with review, not in spontaneous kubectl patch commands.
  • Emergency access: Direct contexts (ACE) and a break-glass account with a sealed password are prepared before Rancher goes down.
  • After every incident: A short blameless report on what happened and what will prevent it from happening again.