Skip to content
Hands-on IT IT Drills

Day 17: Competition – the Sabotaged Cluster

Goal: You find and fix six faults across node, control plane, app, network, storage and RBAC – against the clock. · Duration: approx. 60 min · Prerequisite: Days 0–16

Briefing

You take over a cluster with the namespace arena. The handover note says “it’s running”. In reality, someone has planted six faults. Set a timer for 40 minutes – that corresponds to roughly six exam tasks.

Rules of Play

  • First run the sabotage (in a real competition, someone else does this), then do not page back anymore.
  • Per round: read the situation, find the cause, fix it, prove it with one command.
  • Only read the hints after five minutes without progress.
  • Scoring at the end: one point per round solved without a hint.

Sabotage

On worker2:

sudo systemctl disable --now kubelet

On cp1:

sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/kube-scheduler.yaml.bak
sudo sed -i 's#kube-scheduler:v1.37.2#kube-scheduler:v1.37.99#' \
  /etc/kubernetes/manifests/kube-scheduler.yaml
k create ns arena
cat <<'EOF' | k apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
  namespace: arena
spec:
  replicas: 2
  selector:
    matchLabels:
      app: shop
  template:
    metadata:
      labels:
        app: shop
    spec:
      containers:
      - name: web
        image: nginx:1.29-alpinex
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: shop
  namespace: arena
spec:
  selector:
    app: shopp
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: freeze
  namespace: arena
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: arena
spec:
  storageClassName: fast
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: arena
EOF
k run client -n arena --image=busybox:1.37 -- sleep 3600

Start the clock.

Round 1: A Node Is Missing

Situation: k get nodes shows one node NotReady. It should also be back after a reboot.

Hint: On the node, check not only whether the service is running, but also whether it starts at boot.

Solution

On worker2:

systemctl is-enabled kubelet
systemctl is-active kubelet
sudo systemctl enable --now kubelet

Expected:

disabled
inactive
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service →
/usr/lib/systemd/system/kubelet.service.

Proof on cp1: k get node worker2 shows Ready after about 20 seconds.

Save your progress? Sign up for free

Round 2: The Hidden Fault

Situation: New Pods are being scheduled – yet something is wrong with the control plane. Find the broken component.

Hint: In an HA cluster, another instance takes over – the fault only shows up if you look for it.

Solution

k get pods -n kube-system -l tier=control-plane | grep -v Running
k describe pod -n kube-system kube-scheduler-cp1 | grep -m1 'Failed to pull'

Expected (abridged):

kube-scheduler-cp1   0/1   ErrImagePull   0   51s
  Warning  Failed  43s (x3 over 81s)  kubelet  spec.containers{kube-scheduler}: Failed
to
pull image "registry.k8s.io/kube-scheduler:v1.37.99": rpc error: code = NotFound ...

On cp1:

sudo sed -i 's#kube-scheduler:v1.37.99#kube-scheduler:v1.37.2#' \
  /etc/kubernetes/manifests/kube-scheduler.yaml

Proof: After about 30 seconds, kube-scheduler-cp1 is Running. Leader election had long since handed over to worker1 (k get lease -n kube-system kube-scheduler) – that’s why scheduling kept working.

Save your progress? Sign up for free

Round 3: The Shop Won’t Start

Situation: k get deploy shop -n arena shows 0/2.

Hint: Read the status of the Pods and the events.

Solution

k get pods -n arena -l app=shop
k describe pod -n arena -l app=shop | grep -m1 'Failed to pull'
k set image deployment/shop -n arena web=nginx:1.29-alpine
k rollout status deployment shop -n arena

Expected (abridged):

shop-69b47d6db9-n7vvc   0/1   ErrImagePull   0   62s
...
deployment "shop" successfully rolled out

The tag 1.29-alpinex does not exist. One Pod was additionally stuck on the dead worker2 – Round 1 rescued it along the way.

Save your progress? Sign up for free

Round 4: The Shop Doesn’t Respond

Situation: The Pods are running, but k exec -n arena client -- wget -qO- -T3 shop returns nothing.

Hint: There are two faults. The first shows up in the EndpointSlices, the second in the kind of error message.

Solution

k get endpointslices -n arena -l kubernetes.io/service-name=shop
k get svc shop -n arena -o jsonpath='{.spec.selector}{"\n"}'
k set selector svc shop -n arena app=shop
k exec -n arena client -- wget -qO- -T3 shop

Expected (abridged):

NAME         ADDRESSTYPE   PORTS     ENDPOINTS   AGE
shop-...     IPv4          <unset>   <unset>     2m
{"app":"shopp"}
service/shop selector updated
wget: download timed out
command terminated with exit code 1

Now timed out instead of refused – a NetworkPolicy:

k get netpol -n arena
cat <<'EOF' | k apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: shop-from-arena
  namespace: arena
spec:
  podSelector:
    matchLabels:
      app: shop
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}
    ports:
    - port: 80
EOF
k exec -n arena client -- wget -qO- -T3 shop | grep -o '<title>.*</title>'

Expected:

NAME     POD-SELECTOR   AGE
freeze   <none>         3m
networkpolicy.networking.k8s.io/shop-from-arena created
<title>Welcome to nginx!</title>

freeze stays in place as the baseline; the new policy opens port 80 specifically for Pods from arena.

Save your progress? Sign up for free

Round 5: No Storage

Situation: The PVC data stays Pending.

Hint: storageClassName cannot be changed on an existing PVC.

Solution

k describe pvc data -n arena | tail -1
k get sc
k delete pvc data -n arena
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: arena
spec:
  storageClassName: nfs
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
EOF
k get pvc data -n arena

Expected (abridged):

  Warning  ProvisioningFailed  13s (x5 over 71s)  persistentvolume-controller
storageclass.storage.k8s.io "fast" not found
NAME            PROVISIONER      ...
nfs (default)   nfs.csi.k8s.io   ...
persistentvolumeclaim "data" deleted from arena namespace
persistentvolumeclaim/data created
NAME   STATUS   VOLUME                                     CAPACITY   ACCESS MODES
data   Bound    pvc-99eaef27-09c8-4f60-b24f-70b625cf7833   1Gi        RWO

Save your progress? Sign up for free

Round 6: The Deployer Sees Nothing

Situation: The CI pipeline uses the ServiceAccount deployer in arena and should be allowed to read everything there.

Hint: There is a matching built-in ClusterRole.

Solution

k auth can-i list pods -n arena --as=system:serviceaccount:arena:deployer
k create rolebinding deployer-view -n arena --clusterrole=view \
  --serviceaccount=arena:deployer
k auth can-i list pods -n arena --as=system:serviceaccount:arena:deployer

Expected:

no
rolebinding.rbac.authorization.k8s.io/deployer-view created
yes

Stop the clock.

Save your progress? Sign up for free

Checkpoint

k get nodes

Expected: three nodes Ready.

k get pods -n kube-system -l tier=control-plane --no-headers | grep -vc Running

Expected: 0

k exec -n arena client -- wget -qO- -T3 shop | grep -c 'Welcome to nginx'

Expected: 2 (title and heading of the nginx start page).

k get pvc data -n arena -o jsonpath='{.status.phase}{"\n"}'

Expected: Bound

Scoring

Points Assessment
6 exam-ready – continue with the practice exam (Appendix B)
4–5 good; repeat the days you missed
0–3 work through Days 14 and 15 again

If you stayed under 30 minutes, you have time to spare in the exam.

Challenge

Build your own sabotage for a study partner with three faults from three different days, write the situation on a slip of paper and let them play. Good candidates: a wrong targetPort (Day 15), a NoExecute taint (Day 5), a broken CoreDNS ConfigMap (Day 7).

Hint: Back up every file before you change it – and write down the repair.

Solution

There is no fixed solution. Afterwards, check together with this day’s checkpoint commands whether everything is running again.

Clean up:

k delete ns arena

Save your progress? Sign up for free

Exam Tip

  • First skim all tasks, solve the quick ones first. Flag a task where you are stuck after five minutes and come back to it later.
  • Finish every task with a proof command – that way you notice immediately whether a second fault is still hiding (Round 4).

Enterprise Note

  • In operations, exercises like these are called “game days” or chaos engineering; tools like Chaos Mesh or Litmus automate the sabotage.
  • Every real incident ends with a postmortem: What did the monitoring alert miss?