Day 17: Competition – the Sabotaged Cluster
Goal: You find and fix six faults across node, control plane, app, network, storage and RBAC – against the clock. · Duration: approx. 60 min · Prerequisite: Days 0–16
Briefing
You take over a cluster with the namespace arena. The
handover note says “it’s running”. In reality, someone has planted six
faults. Set a timer for 40 minutes – that corresponds
to roughly six exam tasks.
Rules of Play
- First run the sabotage (in a real competition, someone else does this), then do not page back anymore.
- Per round: read the situation, find the cause, fix it, prove it with one command.
- Only read the hints after five minutes without progress.
- Scoring at the end: one point per round solved without a hint.
Sabotage
On worker2:
sudo systemctl disable --now kubeletOn cp1:
sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/kube-scheduler.yaml.bak
sudo sed -i 's#kube-scheduler:v1.37.2#kube-scheduler:v1.37.99#' \
/etc/kubernetes/manifests/kube-scheduler.yaml
k create ns arena
cat <<'EOF' | k apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop
namespace: arena
spec:
replicas: 2
selector:
matchLabels:
app: shop
template:
metadata:
labels:
app: shop
spec:
containers:
- name: web
image: nginx:1.29-alpinex
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: shop
namespace: arena
spec:
selector:
app: shopp
ports:
- port: 80
targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: freeze
namespace: arena
spec:
podSelector: {}
policyTypes:
- Ingress
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
namespace: arena
spec:
storageClassName: fast
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: deployer
namespace: arena
EOF
k run client -n arena --image=busybox:1.37 -- sleep 3600Start the clock.
Round 1: A Node Is Missing
Situation: k get nodes shows one node
NotReady. It should also be back after a reboot.
Hint: On the node, check not only whether the service is running, but also whether it starts at boot.
Solution
On worker2:
systemctl is-enabled kubelet
systemctl is-active kubelet
sudo systemctl enable --now kubeletExpected:
disabled
inactive
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service →
/usr/lib/systemd/system/kubelet.service.
Proof on cp1: k get node worker2 shows
Ready after about 20 seconds.
Save your progress? Sign up for free
Round 2: The Hidden Fault
Situation: New Pods are being scheduled – yet something is wrong with the control plane. Find the broken component.
Hint: In an HA cluster, another instance takes over – the fault only shows up if you look for it.
Solution
k get pods -n kube-system -l tier=control-plane | grep -v Running
k describe pod -n kube-system kube-scheduler-cp1 | grep -m1 'Failed to pull'Expected (abridged):
kube-scheduler-cp1 0/1 ErrImagePull 0 51s
Warning Failed 43s (x3 over 81s) kubelet spec.containers{kube-scheduler}: Failed
to
pull image "registry.k8s.io/kube-scheduler:v1.37.99": rpc error: code = NotFound ...
On cp1:
sudo sed -i 's#kube-scheduler:v1.37.99#kube-scheduler:v1.37.2#' \
/etc/kubernetes/manifests/kube-scheduler.yamlProof: After about 30 seconds, kube-scheduler-cp1 is
Running. Leader election had long since handed over to
worker1
(k get lease -n kube-system kube-scheduler) – that’s why
scheduling kept working.
Save your progress? Sign up for free
Round 3: The Shop Won’t Start
Situation: k get deploy shop -n arena
shows 0/2.
Hint: Read the status of the Pods and the events.
Solution
k get pods -n arena -l app=shop
k describe pod -n arena -l app=shop | grep -m1 'Failed to pull'
k set image deployment/shop -n arena web=nginx:1.29-alpine
k rollout status deployment shop -n arenaExpected (abridged):
shop-69b47d6db9-n7vvc 0/1 ErrImagePull 0 62s
...
deployment "shop" successfully rolled out
The tag 1.29-alpinex does not exist. One Pod was
additionally stuck on the dead worker2 – Round 1 rescued it
along the way.
Save your progress? Sign up for free
Round 4: The Shop Doesn’t Respond
Situation: The Pods are running, but
k exec -n arena client -- wget -qO- -T3 shop returns
nothing.
Hint: There are two faults. The first shows up in the EndpointSlices, the second in the kind of error message.
Solution
k get endpointslices -n arena -l kubernetes.io/service-name=shop
k get svc shop -n arena -o jsonpath='{.spec.selector}{"\n"}'
k set selector svc shop -n arena app=shop
k exec -n arena client -- wget -qO- -T3 shopExpected (abridged):
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
shop-... IPv4 <unset> <unset> 2m
{"app":"shopp"}
service/shop selector updated
wget: download timed out
command terminated with exit code 1
Now timed out instead of refused – a
NetworkPolicy:
k get netpol -n arena
cat <<'EOF' | k apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: shop-from-arena
namespace: arena
spec:
podSelector:
matchLabels:
app: shop
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {}
ports:
- port: 80
EOF
k exec -n arena client -- wget -qO- -T3 shop | grep -o '<title>.*</title>'Expected:
NAME POD-SELECTOR AGE
freeze <none> 3m
networkpolicy.networking.k8s.io/shop-from-arena created
<title>Welcome to nginx!</title>
freeze stays in place as the baseline; the new policy
opens port 80 specifically for Pods from arena.
Save your progress? Sign up for free
Round 5: No Storage
Situation: The PVC data stays
Pending.
Hint: storageClassName cannot be
changed on an existing PVC.
Solution
k describe pvc data -n arena | tail -1
k get sc
k delete pvc data -n arena
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
namespace: arena
spec:
storageClassName: nfs
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
EOF
k get pvc data -n arenaExpected (abridged):
Warning ProvisioningFailed 13s (x5 over 71s) persistentvolume-controller
storageclass.storage.k8s.io "fast" not found
NAME PROVISIONER ...
nfs (default) nfs.csi.k8s.io ...
persistentvolumeclaim "data" deleted from arena namespace
persistentvolumeclaim/data created
NAME STATUS VOLUME CAPACITY ACCESS MODES
data Bound pvc-99eaef27-09c8-4f60-b24f-70b625cf7833 1Gi RWO
Save your progress? Sign up for free
Round 6: The Deployer Sees Nothing
Situation: The CI pipeline uses the ServiceAccount
deployer in arena and should be allowed to
read everything there.
Hint: There is a matching built-in ClusterRole.
Solution
k auth can-i list pods -n arena --as=system:serviceaccount:arena:deployer
k create rolebinding deployer-view -n arena --clusterrole=view \
--serviceaccount=arena:deployer
k auth can-i list pods -n arena --as=system:serviceaccount:arena:deployerExpected:
no
rolebinding.rbac.authorization.k8s.io/deployer-view created
yes
Stop the clock.
Save your progress? Sign up for free
Checkpoint
k get nodesExpected: three nodes Ready.
k get pods -n kube-system -l tier=control-plane --no-headers | grep -vc RunningExpected: 0
k exec -n arena client -- wget -qO- -T3 shop | grep -c 'Welcome to nginx'Expected: 2 (title and heading of the
nginx start page).
k get pvc data -n arena -o jsonpath='{.status.phase}{"\n"}'Expected: Bound
Scoring
| Points | Assessment |
|---|---|
| 6 | exam-ready – continue with the practice exam (Appendix B) |
| 4–5 | good; repeat the days you missed |
| 0–3 | work through Days 14 and 15 again |
If you stayed under 30 minutes, you have time to spare in the exam.
Challenge
Build your own sabotage for a study partner with three faults from
three different days, write the situation on a slip of paper and let
them play. Good candidates: a wrong targetPort (Day 15), a
NoExecute taint (Day 5), a broken CoreDNS ConfigMap (Day
7).
Hint: Back up every file before you change it – and write down the repair.
Solution
There is no fixed solution. Afterwards, check together with this day’s checkpoint commands whether everything is running again.
Clean up:
k delete ns arenaSave your progress? Sign up for free
Exam Tip
- First skim all tasks, solve the quick ones first. Flag a task where you are stuck after five minutes and come back to it later.
- Finish every task with a proof command – that way you notice immediately whether a second fault is still hiding (Round 4).
Enterprise Note
- In operations, exercises like these are called “game days” or chaos engineering; tools like Chaos Mesh or Litmus automate the sabotage.
- Every real incident ends with a postmortem: What did the monitoring alert miss?