Skip to content
Hands-on IT IT Drills

Day 19: Competition – Securing the Insecure Cluster

Goal: You find and fix eight security holes across the CKS curriculum – against the clock, with proof before and after. · Duration: approx. 90 min · Prerequisite: Days 0–18

Briefing

  • All six curriculum domains: Cluster Setup, Cluster Hardening, System Hardening, Microservice Vulnerabilities, Supply Chain Security, Monitoring/Logging/Runtime Security.
  • Eight rounds; six of them in their own namespace (tag19-r1 … tag19-r8), round 4 (kubelet) and round 5 (API server) plant a hole in the cluster itself – you roll it back.
  • Every round ends with a proof command: insecure before, secure after.
  • The hardening from Days 3, 6, 10, 11, 12, 15 and 16 must still be active at the end (Checkpoint).
Round  Hole                                   Tool
1      privileged Pod with hostPath           PSA restricted, securityContext
2      ServiceAccount with cluster-admin      Role/RoleBinding, auth can-i
3      NetworkPolicy "allow-all"              Default Deny, podSelector
4      kubelet anonymous + AlwaysAllow        /var/lib/kubelet/config.yaml
5      NodeRestriction missing                kube-bench, static Pod manifest
6      plaintext key, Secret reader           secretKeyRef, audit log + jq
7      image with CRITICAL vulnerabilities    Trivy, RuntimeClass gvisor
8      Falco reports /etc/shadow              Falco logs, quarantine
mkdir -p ~/cks/tag19/backup && cd ~/cks/tag19

Rules of Play

  • Per round: create the namespace, run the setup commands, then start the clock. Do not study the manifest – in the exam, too, you only see it once you go looking.
  • Find the cause, fix it, prove it with one command. Only then stop the clock.
  • The task’s security requirement stays in place: whoever switches off a hardening so the Pod runs gets zero points.
  • Only read the Hint once the target time has run out.

Every round with its own namespace starts the same way (here for round 1):

k create ns tag19-r1
k config set-context --current --namespace=tag19-r1

Round 1: Backup with a view of the host

Situation: Team 1 runs a backup container. From now on, the namespace is to enforce the Pod Security Standard restricted, and backup must keep running afterwards – with a read-only root filesystem. /data stays writable.

k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backup
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backup
  template:
    metadata:
      labels:
        app: backup
    spec:
      containers:
      - name: backup
        image: busybox:1.37
        command: ["sh", "-c", "while true; do date > /data/last; sleep 30; done"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /data
        - name: host
          mountPath: /host
      volumes:
      - name: data
        emptyDir: {}
      - name: host
        hostPath:
          path: /
EOF
k rollout status deploy/backup
k exec deploy/backup -- ls /host/etc/kubernetes

Expected:

deployment.apps/backup created
deployment "backup" successfully rolled out
kubelet.conf
manifests
pki
tmp

The container sees the node’s entire filesystem – including the kubelet credentials.

Target time: 4 min

Hint: The label only affects new Pods. The warning you get when setting it is your checklist – six items.

Solution

k label ns tag19-r1 pod-security.kubernetes.io/enforce=restricted
k rollout restart deploy/backup
k get events --field-selector reason=FailedCreate | tail -1

Expected (abridged):

namespace/tag19-r1 labeled
Warning: existing pods in namespace "tag19-r1" violate the new PodSecurity enforce
level "restricted:latest"
Warning: backup-765668c6fb-cmvj4: privileged, allowPrivilegeEscalation != false,
unrestricted capabilities, restricted volume types, runAsNonRoot != true, seccompProfile
deployment.apps/backup restarted
1s   Warning   FailedCreate   replicaset/backup-767fc5b66c   ... Error creating: pods
"backup-767fc5b66c-hlc8x" is forbidden: violates PodSecurity "restricted:latest": ...

The old Pod keeps running, new ones are rejected. All six items plus the read-only root FS in one JSON patch (index 1 = volume host):

k patch deploy backup --type=json -p='[
 {"op": "remove", "path": "/spec/template/spec/volumes/1"},
 {"op": "remove", "path": "/spec/template/spec/containers/0/volumeMounts/1"},
 {"op": "add", "path": "/spec/template/spec/securityContext", "value":
  {"runAsNonRoot": true, "runAsUser": 65532,
   "seccompProfile": {"type": "RuntimeDefault"}}},
 {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext",
  "value": {"allowPrivilegeEscalation": false, "readOnlyRootFilesystem": true,
   "capabilities": {"drop": ["ALL"]}}}]'
k rollout status deploy/backup
k exec deploy/backup -- id
k exec deploy/backup -- touch /x
k exec deploy/backup -- sh -c 'date > /data/test && echo ok'
k label --dry-run=server --overwrite ns tag19-r1 \
  pod-security.kubernetes.io/enforce=restricted

Expected (abridged):

deployment.apps/backup patched
deployment "backup" successfully rolled out
uid=65532 gid=0(root) groups=0(root)
touch: /x: Read-only file system
command terminated with exit code 1
ok
namespace/tag19-r1 not labeled (server dry run)

The server dry run without a warning proves it: no running Pod violates restricted anymore. restricted does not require readOnlyRootFilesystem – that was the extra requirement.

Save your progress? Sign up for free

Round 2: The deployer is cluster admin

Situation: The pipeline uses the ServiceAccount deployer in tag19-r2. It may read and change Deployments in this namespace (get, list, watch, update, patch) – nothing else.

k create ns tag19-r2
k config set-context --current --namespace=tag19-r2
k create sa deployer
k create clusterrolebinding tag19-r2-deployer --clusterrole=cluster-admin \
  --serviceaccount=tag19-r2:deployer

Target time: 3 min

Hint: The binding is not in the namespace – k get rolebindings stays empty.

Solution

SA=system:serviceaccount:tag19-r2:deployer
k auth can-i get secrets -A --as=$SA
k get clusterrolebindings -o wide | grep -E 'NAME|deployer'

Expected (abridged):

yes
NAME                ROLE                        AGE  ...  SERVICEACCOUNTS
tag19-r2-deployer   ClusterRole/cluster-admin   1s   ...  tag19-r2/deployer

Delete the binding, create a narrow Role:

k delete clusterrolebinding tag19-r2-deployer
k create role deployer --verb=get,list,watch,update,patch \
  --resource=deployments.apps
k create rolebinding deployer --role=deployer --serviceaccount=tag19-r2:deployer
k auth can-i patch deployments -n tag19-r2 --as=$SA
k auth can-i create deployments -n tag19-r2 --as=$SA
k auth can-i get secrets -n tag19-r2 --as=$SA
k auth can-i patch deployments -n kube-system --as=$SA

Expected:

clusterrolebinding.rbac.authorization.k8s.io "tag19-r2-deployer" deleted
role.rbac.authorization.k8s.io/deployer created
rolebinding.rbac.authorization.k8s.io/deployer created
yes
no
no
no

Four can-i prove both directions: what should be allowed works; creating, Secrets and other namespaces do not.

Save your progress? Sign up for free

Round 3: The policy that allows everything

Situation: notes-api runs in tag19-r3. Only Pods with the label role=frontend from the same namespace may access port 8080; all other incoming traffic into the namespace is to be blocked.

k create ns tag19-r3
k config set-context --current --namespace=tag19-r3
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  selector:
    app: notes
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all
spec:
  podSelector: {}
  ingress:
  - {}
EOF
k run frontend --image=busybox:1.37 -l role=frontend -- sleep 3600
k run fremd --image=busybox:1.37 -- sleep 3600
k create ns tag19-r3-gast
k -n tag19-r3-gast run gast --image=busybox:1.37 -l role=frontend -- sleep 3600

Target time: 4 min

Hint: NetworkPolicies add up. A deny policy next to one that allows everything changes nothing.

Solution

Situation – all three get through:

k exec frontend -- wget -qO- -T3 notes/healthz
k exec fremd -- wget -qO- -T3 notes/healthz
k -n tag19-r3-gast exec gast -- wget -qO- -T3 notes.tag19-r3/healthz

Expected:

ok
ok
ok

Default deny and the targeted allow:

k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: notes-from-frontend
spec:
  podSelector:
    matchLabels:
      app: notes
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - port: 8080
EOF
k exec fremd -- wget -qO- -T3 notes/healthz
k get netpol

Expected:

networkpolicy.networking.k8s.io/default-deny created
networkpolicy.networking.k8s.io/notes-from-frontend created
ok
NAME                  POD-SELECTOR   AGE
allow-all             <none>         12s
default-deny          <none>         0s
notes-from-frontend   app=notes      0s

fremd still gets through. The reason is in allow-all:

k describe netpol allow-all | sed -n '/Spec/,$p'
k delete netpol allow-all
k exec frontend -- wget -qO- -T3 notes/healthz
k exec fremd -- wget -qO- -T3 notes/healthz
k -n tag19-r3-gast exec gast -- wget -qO- -T3 notes.tag19-r3/healthz

Expected:

Spec:
  PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From: <any> (traffic not restricted by source)
  Not affecting egress traffic
  Policy Types: Ingress
networkpolicy.networking.k8s.io "allow-all" deleted from tag19-r3 namespace
ok
wget: download timed out
command terminated with exit code 1
wget: download timed out
command terminated with exit code 1

gast does carry role=frontend, but it comes from another namespace: a podSelector without a namespaceSelector only applies within its own namespace.

Save your progress? Sign up for free

Round 4: The open kubelet

Situation: “For debugging”, someone opened the kubelet API on worker2. Find the hole from cp1 and close it. Setup from cp1 (backup first):

ssh [email protected]
sudo cp /var/lib/kubelet/config.yaml /root/kubelet-config.tag19
sudo sed -i -e '/anonymous:/{n;s/enabled: false/enabled: true/}' \
  -e 's/  mode: Webhook/  mode: AlwaysAllow/' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
exit

Target time: 4 min

Hint: Test port 10250 on all nodes without authentication. ps on the kubelet process reveals the configuration file.

Solution

On cp1:

curl -sk https://172.16.0.228:10250/pods; echo
curl -sk https://172.16.0.83:10250/pods | jq -r '.items[].metadata.name' | head -2
P=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name | cut -d/ -f2)
curl -sk -X POST "https://172.16.0.83:10250/run/kube-system/$P/cilium-agent?cmd=id"

Expected (abridged):

Unauthorized
cilium-zg9q2
falco-ld8gg
uid=0(root) gid=0(root) groups=0(root)

worker1 refuses, worker2 runs commands anonymously as root. On worker2:

ssh [email protected]
ps -o args= -C kubelet | tr ' ' '\n' | grep -- --config
sudo grep -A1 -E "  anonymous:|^authorization:" /var/lib/kubelet/config.yaml
sudo sed -i -e '/anonymous:/{n;s/enabled: true/enabled: false/}' \
  -e 's/  mode: AlwaysAllow/  mode: Webhook/' /var/lib/kubelet/config.yaml
sudo grep -A1 -E "  anonymous:|^authorization:" /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubelet
sudo cmp /var/lib/kubelet/config.yaml /root/kubelet-config.tag19 && echo gleich
sudo rm /root/kubelet-config.tag19
exit

Expected:

--config=/var/lib/kubelet/config.yaml
  anonymous:
    enabled: true
--
authorization:
  mode: AlwaysAllow
  anonymous:
    enabled: false
--
authorization:
  mode: Webhook
active
gleich

Proof on cp1:

curl -sk https://172.16.0.83:10250/pods; echo
k get nodes worker2

Expected (abridged):

Unauthorized
NAME      STATUS   ROLES    AGE    VERSION
worker2   Ready    <none>   112m   …

cmp against the backup shows: the file is bit-for-bit as it was before the setup. The permissions 600 from Day 3 stay, because sed -i keeps them.

Save your progress? Sign up for free

Round 5: The node is allowed too much

Situation: An audit reports: a compromised worker could give itself labels with the prefix node-restriction.kubernetes.io/ and thus attract security-relevant Pods. Find the cause on the API server and fix it. Setup on cp1 (backup outside the manifest folder):

M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag19/backup/kube-apiserver.yaml
sudo sed -i '/--enable-admission-plugins=NodeRestriction/d' $M/kube-apiserver.yaml

Wait one to two minutes until k get --raw /readyz reports ok again.

Target time: 5 min

Hint: Test with the kubelet’s kubeconfig on worker1. Which CIS check covers admission plugins?

Solution

Proof on worker1 (from cp1 via ssh [email protected]):

export KUBECONFIG=/etc/kubernetes/kubelet.conf
sudo -E kubectl auth whoami | grep Username
sudo -E kubectl label node worker1 node-restriction.kubernetes.io/tag19=pwn
exit

Expected:

Username                                            system:node:worker1
node/worker1 labeled

NodeRestriction would have forbidden that. On cp1:

sudo kube-bench run --targets master --benchmark cis-1.12 --check 1.2.14 \
  | grep -E '^\[(FAIL|PASS)'
sudo grep -c admission $M/kube-apiserver.yaml

Expected:

[FAIL] 1.2.14 Ensure that the admission control plugin NodeRestriction is set
(Automated)
0

Add the flag back – in the exam with vim, here via sed directly after --client-ca-file:

sudo sed -i '/- --client-ca-file=/a\    - --enable-admission-plugins=NodeRestriction' \
  $M/kube-apiserver.yaml
sudo grep -B1 -A1 -- '--enable-admission' $M/kube-apiserver.yaml
sudo md5sum $M/kube-apiserver.yaml ~/cks/tag19/backup/kube-apiserver.yaml

Expected (checksums abridged):

    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
…  /etc/kubernetes/manifests/kube-apiserver.yaml
…  /home/ubuntu/cks/tag19/backup/kube-apiserver.yaml

Both checksums are identical. After the restart (1–2 min):

k get --raw /readyz; echo
ps -o args= -C kube-apiserver | tr ' ' '\n' | grep admission

Expected:

ok
--enable-admission-plugins=NodeRestriction

Proof on worker1, then remove the smuggled-in label as admin on cp1:

export KUBECONFIG=/etc/kubernetes/kubelet.conf
sudo -E kubectl label node worker1 node-restriction.kubernetes.io/tag19=pwn2 \
  --overwrite
exit
k label node worker1 node-restriction.kubernetes.io/tag19-

Expected:

Error from server (Forbidden): nodes "worker1" is forbidden: is not allowed to modify
labels: node-restriction.kubernetes.io/tag19
node/worker1 unlabeled

kubeadm sets the flag on its own (built-in default, not visible in kubeadm-config) – so an upgrade restores it. A label the attacker has set, on the other hand, stays until you remove it.

Save your progress? Sign up for free

Round 6: Plaintext key, Secret with an onlooker

Situation: In tag19-r6, the notes-api API key sits in plaintext in the Deployment. Put it into a Secret notes-key. In addition, the audit log is to show who read the Secret db-cred – you take away that account’s access to Secrets (it may still read ConfigMaps).

k create ns tag19-r6
k config set-context --current --namespace=tag19-r6
k create deployment notes --image=172.16.0.126:30500/notes-api:1.0
k set env deploy/notes NOTES_API_KEY=lab-key-123
k create secret generic db-cred --from-literal=password=Lab-Geheim-19
k create sa reporter
k create role reporter --verb=get,list --resource=secrets,configmaps
k create rolebinding reporter --role=reporter --serviceaccount=tag19-r6:reporter
T=$(k create token reporter --duration=10m)
curl -s --cacert /etc/kubernetes/pki/ca.crt -H "Authorization: Bearer $T" \
  https://172.16.0.126:6443/api/v1/namespaces/tag19-r6/secrets/db-cred > /dev/null

Lab-Geheim-19 is a pure lab value.

Target time: 5 min

Hint: The audit log is in /var/log/kubernetes/audit/audit.log (Day 15); objectRef and user are the fields for jq.

Solution

k get deploy notes -o jsonpath='{.spec.template.spec.containers[0].env}'; echo
L=/var/log/kubernetes/audit/audit.log
sudo jq -c 'select(.objectRef.resource=="secrets" and .objectRef.name=="db-cred"
  and .objectRef.namespace=="tag19-r6" and .verb=="get")
  | [.user.username, .verb, .responseStatus.code, .sourceIPs[0]]' $L
k get rolebindings -o wide

Expected (abridged):

[{"name":"NOTES_API_KEY","value":"lab-key-123"}]
["system:serviceaccount:tag19-r6:reporter","get",200,"172.16.0.126"]
NAME       ROLE            AGE   USERS   GROUPS   SERVICEACCOUNTS
reporter   Role/reporter   7s                     tag19-r6/reporter

Key into the Secret, variable from the Secret, Role without Secrets:

k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123
k set env deploy/notes NOTES_API_KEY-
k set env deploy/notes --from=secret/notes-key
k rollout status deploy/notes
k set env deploy/notes --list
k create role reporter --verb=get,list --resource=configmaps $do | k apply -f -
SA=system:serviceaccount:tag19-r6:reporter
k auth can-i get secrets --as=$SA
k auth can-i list configmaps --as=$SA
k exec deploy/notes -- wget -qO- localhost:8080/healthz; echo

Expected (abridged):

secret/notes-key created
deployment.apps/notes env updated
deployment.apps/notes env updated
deployment "notes" successfully rolled out
# Deployment notes, container notes-api
# NOTES_API_KEY from secret notes-key, key NOTES_API_KEY
Warning: resource roles/reporter is missing the kubectl.kubernetes.io/last-applied-...
role.rbac.authorization.k8s.io/reporter configured
no
yes
ok

The warning is harmless: apply adds the missing annotation. Thanks to Day 10, the new Secret is stored encrypted in etcd (k8s:enc:aescbc:v1:key2, see Checkpoint).

Save your progress? Sign up for free

Round 7: The old proxy image

Situation: No image with CRITICAL vulnerabilities may run in tag19-r7. Replace affected images with nginx:1.29-alpine-slim. Because the proxy processes untrusted data, it should also run in the gVisor sandbox (RuntimeClass gvisor, Day 11).

k create ns tag19-r7
k config set-context --current --namespace=tag19-r7
k create deployment proxy --image=nginx:1.19 --port=80
k expose deploy proxy --port=80
k create deployment notes --image=172.16.0.126:30500/notes-api:1.0
k set env deploy/notes NOTES_API_KEY=lab-key-123

Target time: 5 min (Trivy takes a while per image)

Hint: First get the image list from the Pods, then scan each image individually; --exit-code 1 saves you reading the table.

Solution

k get pods -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' \
  | sort -u
trivy image -q --severity CRITICAL nginx:1.19 | grep Total
trivy image -q --insecure --severity CRITICAL --exit-code 1 \
  172.16.0.126:30500/notes-api:1.0 > /dev/null; echo $?
trivy image -q --severity CRITICAL --exit-code 1 nginx:1.29-alpine-slim \
  > /dev/null; echo $?

Expected:

172.16.0.126:30500/notes-api:1.0
nginx:1.19
Total: 42 (CRITICAL: 42)
0
0

Only the proxy is affected; the replacement image is clean (Trivy DB as of 2026-11-18). Swap the image and move it into the sandbox:

k set image deploy/proxy nginx=nginx:1.29-alpine-slim
k patch deploy proxy -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
k rollout status deploy/proxy
k get deploy -o custom-columns=NAME:.metadata.name,\
IMAGE:.spec.template.spec.containers[0].image,RC:.spec.template.spec.runtimeClassName
k exec deploy/proxy -- dmesg | head -1
k exec deploy/notes -- wget -qO- -S -T5 proxy 2>&1 | grep -E 'HTTP|Server'

Expected (abridged):

deployment.apps/proxy image updated
deployment.apps/proxy patched
deployment "proxy" successfully rolled out
NAME    IMAGE                              RC
notes   172.16.0.126:30500/notes-api:1.0   <none>
proxy   nginx:1.29-alpine-slim             gvisor
[   0.000000] Starting gVisor...
  HTTP/1.1 200 OK
  Server: nginx/1.29.8

The container name nginx comes from k create deployment (name of the image). The RuntimeClass brings its own nodeSelector sandbox=gvisor – the Pod only lands on prepared workers.

Save your progress? Sign up for free

Round 8: Falco raises the alarm

Situation: For minutes, Falco has been reporting access to /etc/shadow. Find the Pod and its owner, cut the Pod off from the network, save YAML and logs to ~/cks/tag19/evidence-r8/, remove it and make sure it does not come back. web keeps running unchanged.

k create ns tag19-r8
k config set-context --current --namespace=tag19-r8
k create deployment web --image=nginx:1.29-alpine-slim --replicas=2
k create deployment sync --image=busybox:1.37 -- \
  sh -c 'while true; do cat /etc/shadow > /dev/null; sleep 20; done'

Wait a minute so Falco has something to report.

Target time: 5 min

Hint: Falco appends k8s_pod_name and k8s_ns_name to every alert. You get the logs of all three Falco Pods with a label selector.

Solution

k -n falco logs -l app.kubernetes.io/name=falco -c falco --since=5m --tail=-1 \
  | grep 'Sensitive file' \
  | grep -o 'file=[^ ]*\|k8s_pod_name=[^ ]*\|k8s_ns_name=[^ ]*' | sort | uniq -c

Expected:

      3 file=/etc/shadow
      3 k8s_ns_name=tag19-r8
      3 k8s_pod_name=sync-7cdd59b4db-gvfwc

Don’t get confused: alerts “Detected ptrace PTRACE_ATTACH attempt” with containerd-shim as the parent process come from starting the gVisor sandbox in round 7.

P=sync-7cdd59b4db-gvfwc
k get pod $P -o jsonpath='{.metadata.ownerReferences[0].name}{"\n"}'
k get pod $P -o jsonpath='{.spec.containers[0].command}{"\n"}'

Expected:

sync-7cdd59b4db
["sh","-c","while true; do cat /etc/shadow > /dev/null; sleep 20; done"]

First the quarantine policy, then detach the Pod from the ReplicaSet, then shut down the owner:

k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine
spec:
  podSelector:
    matchLabels:
      quarantine: "true"
  policyTypes:
  - Ingress
  - Egress
EOF
k label pod $P app- quarantine=true
k scale deploy sync --replicas=0
k get pods --show-labels
k get cep $P -o jsonpath='{.status.identity.labels}' \
  | grep -o 'k8s:quarantine=true'

Empty? Wait briefly and repeat (the identity follows the labels with a delay, Day 18). Then:

k exec $P -- wget -qO- -T3 http://web

Expected (abridged):

networkpolicy.networking.k8s.io/quarantine created
pod/sync-7cdd59b4db-gvfwc modified
deployment.apps/sync scaled
NAME                    READY   STATUS        ...   LABELS
sync-7cdd59b4db-f7pjp   0/1     Terminating   ...   app=sync,pod-template-hash=...
sync-7cdd59b4db-gvfwc   1/1     Running       ...   ...,quarantine=true
web-846db866-q5dll      1/1     Running       ...   app=web,pod-template-hash=846db866
web-846db866-q99rb      1/1     Running       ...   app=web,pod-template-hash=846db866
wget: bad address 'web'
command terminated with exit code 1

The Terminating Pod is the replacement the ReplicaSet started immediately after the relabeling – scale cleans it up. bad address: even DNS is blocked for the quarantined Pod. Secure the evidence, then delete:

mkdir -p ~/cks/tag19/evidence-r8
k get pod $P -o yaml > ~/cks/tag19/evidence-r8/pod.yaml
k logs $P > ~/cks/tag19/evidence-r8/log.txt
k delete pod $P --now
k get deploy

Expected (abridged):

pod "sync-7cdd59b4db-gvfwc" deleted from tag19-r8 namespace
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
sync   0/0     0            0           71s
web    2/2     2            2           71s

One minute later:

k -n falco logs -l app.kubernetes.io/name=falco -c falco --since=30s --tail=-1 \
  | grep -c tag19-r8

Expected: 0

Stop the clock.

Save your progress? Sign up for free

Scoring

Round Hole Target time Points
1 privileged + hostPath, PSA restricted 4 min 2
2 cluster-admin for ServiceAccount 3 min 2
3 NetworkPolicy allow-all 4 min 2
4 kubelet anonymous + AlwaysAllow 4 min 2
5 NodeRestriction missing 5 min 2
6 plaintext key + audit trail 5 min 3
7 CRITICAL image + gVisor 5 min 2
8 isolate the Falco finding 5 min 3

You only get full points within the target time, without a hint and with a proof command; otherwise half (round down). Whoever switches off a hardening gets 0 for the round. Overall target time: 35 min, maximum: 18 points.

Points Assessment
15–18 exam-ready – move on to the Practice Exam (Appendix B)
10–14 good; repeat the days of the rounds you missed
0–9 work through Days 5, 6, 9 and 15 once more

Checkpoint

First the rounds, then the hardening that stays beyond this day.

k get deploy -A | grep tag19

Expected: backup 1/1 (r1), notes 1/1 (r3, r6, r7), proxy 1/1 (r7), sync 0/0 and web 2/2 (r8).

k get ns -l pod-security.kubernetes.io/enforce=restricted

Expected: tag19-r1 Active …

k auth can-i get secrets -A --as=system:serviceaccount:tag19-r2:deployer

Expected: no

curl -sk https://172.16.0.83:10250/pods; echo

Expected: Unauthorized (likewise for 172.16.0.228)

ssh [email protected] 'sudo stat -c "%a %n" /var/lib/kubelet/config.yaml'

Expected: 600 /var/lib/kubelet/config.yaml

ps -o args= -C kube-apiserver | tr ' ' '\n' \
  | grep -E 'admission|profiling|authentication-config|encryption-pro|audit-log-pa'

Expected:

--enable-admission-plugins=NodeRestriction
--profiling=false
--authentication-config=/etc/kubernetes/authn/authn-config.yaml
--encryption-provider-config=/etc/kubernetes/enc/enc.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
ps -o args= -C kube-scheduler | tr ' ' '\n' | grep profiling

Expected: --profiling=false (likewise for kube-controller-manager)

curl -sk https://172.16.0.126:6443/version | jq -r .code

Expected: 401

P=/etc/kubernetes/pki
E="--cacert=$P/etcd/ca.crt --cert=$P/apiserver-etcd-client.crt"
E="$E --key=$P/apiserver-etcd-client.key"
sudo etcdctl $E get /registry/secrets/tag19-r6/notes-key --print-value-only \
  | hexdump -C | head -2

Expected:

00000000  6b 38 73 3a 65 6e 63 3a  61 65 73 63 62 63 3a 76  |k8s:enc:aescbc:v|
00000010  31 3a 6b 65 79 32 3a 66  bb 42 01 70 b6 e3 2b 65  |1:key2:f.B.p..+e|
k -n falco get ds falco
k get runtimeclass gvisor

Expected: falco with 3 under DESIRED and READY; gvisor with handler runsc.

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status

Expected (abridged):

Encryption: Wireguard
Interface: cilium_wg0
    Number of peers: 2
k get --raw='/readyz?verbose' | tail -1
k get nodes
k get pods -A --no-headers | grep -vc Running

Expected: readyz check passed, three nodes Ready, 0.

Challenge

A colleague has hidden two leftovers in the cluster. Find them cluster-wide, without knowing the namespace, and remove them. Legitimate system Pods (Falco needs privileged) stay.

k create ns tag19-ch
k -n tag19-ch create sa ci
k create clusterrolebinding tag19-ch-ci --clusterrole=cluster-admin \
  --serviceaccount=tag19-ch:ci
k -n tag19-ch run debug --image=busybox:1.37 --privileged -- sleep 3600

Hint: Two jq queries: Pods with privileged or hostPath outside of kube-system, and all bindings to cluster-admin.

Solution

k get pods -A -o json | jq -r '.items[]
  | select(.metadata.namespace != "kube-system")
  | select(any(.spec.containers[]; .securityContext.privileged == true)
      or any(.spec.volumes[]?; .hostPath))
  | .metadata.namespace + "/" + .metadata.name'
k get clusterrolebindings -o json | jq -r '.items[]
  | select(.roleRef.name == "cluster-admin")
  | .metadata.name + " " + ([.subjects[]? | .kind + ":" + .name] | join(","))'

Expected:

falco/falco-j78wr
falco/falco-ld8gg
falco/falco-vlb49
tag19-ch/debug
cluster-admin Group:system:masters
kubeadm:cluster-admins Group:kubeadm:cluster-admins
tag19-ch-ci ServiceAccount:ci

Falco and kubeadm’s two group bindings are intended; debug and tag19-ch-ci are not:

k -n tag19-ch delete pod debug --now
k delete clusterrolebinding tag19-ch-ci
k auth can-i '*' '*' --as=system:serviceaccount:tag19-ch:ci

Expected:

pod "debug" deleted from tag19-ch namespace
clusterrolebinding.rbac.authorization.k8s.io "tag19-ch-ci" deleted
no

Clean up (the cluster changes from rounds 4 and 5 have already been rolled back):

k config set-context --current --namespace=default
k delete ns tag19-r1 tag19-r2 tag19-r3 tag19-r3-gast tag19-r6 tag19-r7 tag19-r8 \
  tag19-ch

Save your progress? Sign up for free

Exam Tip

  • Finish every task with before/after proof (auth can-i, wget, curl on 10250, server dry run): that way hidden second errors such as allow-all (round 3) or old Pods (round 1) stand out.
  • Before every change to a static Pod manifest or the kubelet configuration: a copy outside of /etc/kubernetes/manifests, afterwards readyz and k get nodes. An API server that doesn’t come back costs more points than the task is worth.
  • Never weaken the security requirement to get a Pod running: adapt the Pod to the requirement (round 1), not the requirement to the Pod.

Enterprise Note

  • Policy as code prevents almost all of today’s holes before deployment: PSA labels via namespace templates, Kyverno/Gatekeeper or ValidatingAdmissionPolicy against privileged, hostPath, foreign registries and cluster-admin bindings.
  • Node and API server configuration belongs in configuration management (Ansible, Cluster API, image pipelines) with drift detection; kube-bench as a Job.
  • Image scans belong in CI and continuously in the cluster (e.g. Trivy Operator), because new CVEs make old images critical overnight.
  • Falco alerts go to the SIEM and on-call via Falcosidekick; quarantine policy and evidence collection belong in the incident runbook, not in someone’s head.
  • Regular “red team days” like this Competition show whether alerts, runbooks and responsibilities hold up when it counts.