Skip to content
Hands-on IT IT Drills

Day 15: Competition – the Bugged Cluster

Goal: You find and close five misconfigurations in someone else’s namespace against the clock. · Duration: approx. 75 min · Prerequisite: Days 1–14

Briefing

Game situation: You take over the namespace arena – supposedly “fully rolled out”. In reality, it contains five of the mistakes from this book. Your task: find them all, close them all, then get the final check green. Set yourself a timer: 30 minutes.

First, let the namespace get bugged (in a real competition, someone else does this):

kubectl create namespace arena
kubectl -n arena create secret generic payments --from-literal=key=sk-live-LAB

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: debug, namespace: arena}
spec:
  containers:
  - {name: c, image: docker.io/library/busybox:1.36, command: ["sleep","infinity"],
     securityContext: {privileged: true}}
---
apiVersion: v1
kind: Pod
metadata: {name: backup, namespace: arena}
spec:
  containers:
  - {name: c, image: docker.io/library/busybox:1.36, command: ["sleep","infinity"],
     volumeMounts: [{name: h, mountPath: /host}]}
  volumes: [{name: h, hostPath: {path: /}}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: arena}
spec:
  replicas: 1
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - {name: c, image: nginxinc/nginx-unprivileged:stable,
         env: [{name: DB_PASSWORD, value: p4ss-lab}]}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: arena-secret-reader}
rules: [{apiGroups: [""], resources: ["secrets"], verbs: ["get","list"]}]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: {name: arena-secret-reader}
subjects: [{kind: ServiceAccount, name: default, namespace: arena}]
roleRef: {kind: ClusterRole, name: arena-secret-reader, apiGroup: rbac.authorization.k8s.io}
EOF
kubectl -n arena expose deployment web --type=NodePort --port=80 --name=web-np

Reconnaissance – find the five findings

Run through your audit commands from the previous days:

# Privileged pods (Day 5)
kubectl -n arena get pods -o json | python3 -c '
import sys,json
[print("privileged:",p["metadata"]["name"]) for p in json.load(sys.stdin)["items"]
 for c in p["spec"]["containers"] if (c.get("securityContext") or {}).get("privileged")]'

# hostPath (Day 6)
kubectl -n arena get pods -o json | python3 -c '
import sys,json
[print("hostPath:",p["metadata"]["name"]) for p in json.load(sys.stdin)["items"]
 for v in (p["spec"].get("volumes") or []) if v.get("hostPath")]'

# Plaintext secret env (Day 10)
kubectl -n arena get deploy -o json | python3 -c '
import sys,json
[print("env-secret:",d["metadata"]["name"],e["name"]) for d in json.load(sys.stdin)["items"]
 for c in d["spec"]["template"]["spec"]["containers"] for e in (c.get("env") or [])
 if "value" in e and any(x in e["name"].upper() for x in ("PASS","KEY","SECRET","TOKEN"))]'

# Overly broad RBAC (Day 2)
kubectl auth can-i list secrets --as=system:serviceaccount:arena:default -A

# NodePort without auth (Day 9)
kubectl get svc -A --field-selector spec.type=NodePort --no-headers | awk '{print $1,$2}'

Expected (your five findings):

privileged: debug
hostPath: backup
env-secret: web DB_PASSWORD
yes
arena web-np

The solution – close everything

# Finding 1+2: privileged & hostPath – Pod Security Admission restricted (Day 5/6)
kubectl label namespace arena pod-security.kubernetes.io/enforce=restricted --overwrite
kubectl -n arena delete pod debug backup

# Finding 3: plaintext env -> Secret + secretKeyRef, and make web PSA-compliant (Day 5/10)
kubectl -n arena create secret generic web-db --from-literal=password=p4ss-lab
kubectl -n arena patch deploy web --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/env",
   "value":[{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":
     {"name":"web-db","key":"password"}}}]},
  {"op":"add","path":"/spec/template/spec/containers/0/securityContext",
   "value":{"allowPrivilegeEscalation":false,"runAsNonRoot":true,
     "capabilities":{"drop":["ALL"]},"seccompProfile":{"type":"RuntimeDefault"}}}]'

# Finding 4: remove overly broad RBAC (Day 2)
kubectl delete clusterrolebinding arena-secret-reader
kubectl delete clusterrole arena-secret-reader

# Finding 5: NodePort -> gone (reachable internally via ClusterIP) (Day 9)
kubectl -n arena delete svc web-np

# Additionally: default deny as the baseline (Day 8)
kubectl -n arena apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-ingress, namespace: arena}
spec: {podSelector: {}, policyTypes: ["Ingress"]}
EOF

Checkpoint – the final check

Every line must be clean:

echo -n "privileged: "; kubectl -n arena get pods -o json | python3 -c '
import sys,json;print(sum(1 for p in json.load(sys.stdin)["items"]
 for c in p["spec"]["containers"] if (c.get("securityContext") or {}).get("privileged")))'
echo -n "hostPath:    "; kubectl -n arena get pods -o json | python3 -c '
import sys,json;print(sum(1 for p in json.load(sys.stdin)["items"]
 for v in (p["spec"].get("volumes") or []) if v.get("hostPath")))'
echo -n "env-secret:  "; kubectl -n arena get deploy -o json | python3 -c '
import sys,json;print(sum(1 for d in json.load(sys.stdin)["items"]
 for c in d["spec"]["template"]["spec"]["containers"] for e in (c.get("env") or [])
 if "value" in e and any(x in e["name"].upper() for x in ("PASS","KEY","SECRET","TOKEN"))))'
echo -n "rbac secrets: "; kubectl auth can-i list secrets \
  --as=system:serviceaccount:arena:default -A
echo -n "nodeports:   "; kubectl -n arena get svc --field-selector spec.type=NodePort \
  --no-headers 2>/dev/null | wc -l

Expected:

privileged: 0
hostPath:    0
env-secret:  0
rbac secrets: no
nodeports:   0

Five out of five. Clear out the arena:

kubectl delete namespace arena

Challenge

Bug the arena a second time yourself – add a sixth finding that this book covers (e.g. a Service with externalIPs from Day 13, or a pod entirely without automountServiceAccountToken: false that reaches the API). Let a colleague search for it.

Hint: A good finding is quiet – it looks like normal configuration and does exactly one step too much.

Solution

There is no single solution – that is the point. Whoever builds findings understands them best. Take the audit commands from the cheat sheet (Appendix A) as your check list and extend it with every new finding.

Save your progress? Sign up for free

Enterprise Note

  • Run such “gamedays” regularly: one team bugs, one team hardens against the clock.
  • Pour the audit commands into a CI job or a cron job – then the machine finds the findings before an attacker does.
  • The best state is one in which most findings cannot arise in the first place: PSA defaults, default-deny network policies and Kyverno guardrails enforce this from the start (Days 5–14).
  • Document every finding with “detected like this, closed like this, prevented like this” – that is the handover that holds up in operations.