Day 19: Competition – Securing the Insecure Cluster
Goal: You find and fix eight security holes across the CKS curriculum – against the clock, with proof before and after. · Duration: approx. 90 min · Prerequisite: Days 0–18
Briefing
- All six curriculum domains: Cluster Setup, Cluster Hardening, System Hardening, Microservice Vulnerabilities, Supply Chain Security, Monitoring/Logging/Runtime Security.
- Eight rounds; six of them in their own namespace
(
tag19-r1…tag19-r8), round 4 (kubelet) and round 5 (API server) plant a hole in the cluster itself – you roll it back. - Every round ends with a proof command: insecure before, secure after.
- The hardening from Days 3, 6, 10, 11, 12, 15 and 16 must still be active at the end (Checkpoint).
Round Hole Tool
1 privileged Pod with hostPath PSA restricted, securityContext
2 ServiceAccount with cluster-admin Role/RoleBinding, auth can-i
3 NetworkPolicy "allow-all" Default Deny, podSelector
4 kubelet anonymous + AlwaysAllow /var/lib/kubelet/config.yaml
5 NodeRestriction missing kube-bench, static Pod manifest
6 plaintext key, Secret reader secretKeyRef, audit log + jq
7 image with CRITICAL vulnerabilities Trivy, RuntimeClass gvisor
8 Falco reports /etc/shadow Falco logs, quarantine
mkdir -p ~/cks/tag19/backup && cd ~/cks/tag19Rules of Play
- Per round: create the namespace, run the setup commands, then start the clock. Do not study the manifest – in the exam, too, you only see it once you go looking.
- Find the cause, fix it, prove it with one command. Only then stop the clock.
- The task’s security requirement stays in place: whoever switches off a hardening so the Pod runs gets zero points.
- Only read the Hint once the target time has run out.
Every round with its own namespace starts the same way (here for round 1):
k create ns tag19-r1
k config set-context --current --namespace=tag19-r1Round 1: Backup with a view of the host
Situation: Team 1 runs a backup container. From now
on, the namespace is to enforce the Pod Security Standard
restricted, and backup must keep running
afterwards – with a read-only root filesystem. /data stays
writable.
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: backup
spec:
replicas: 1
selector:
matchLabels:
app: backup
template:
metadata:
labels:
app: backup
spec:
containers:
- name: backup
image: busybox:1.37
command: ["sh", "-c", "while true; do date > /data/last; sleep 30; done"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /data
- name: host
mountPath: /host
volumes:
- name: data
emptyDir: {}
- name: host
hostPath:
path: /
EOF
k rollout status deploy/backup
k exec deploy/backup -- ls /host/etc/kubernetesExpected:
deployment.apps/backup created
deployment "backup" successfully rolled out
kubelet.conf
manifests
pki
tmp
The container sees the node’s entire filesystem – including the kubelet credentials.
Target time: 4 min
Hint: The label only affects new Pods. The warning you get when setting it is your checklist – six items.
Solution
k label ns tag19-r1 pod-security.kubernetes.io/enforce=restricted
k rollout restart deploy/backup
k get events --field-selector reason=FailedCreate | tail -1Expected (abridged):
namespace/tag19-r1 labeled
Warning: existing pods in namespace "tag19-r1" violate the new PodSecurity enforce
level "restricted:latest"
Warning: backup-765668c6fb-cmvj4: privileged, allowPrivilegeEscalation != false,
unrestricted capabilities, restricted volume types, runAsNonRoot != true, seccompProfile
deployment.apps/backup restarted
1s Warning FailedCreate replicaset/backup-767fc5b66c ... Error creating: pods
"backup-767fc5b66c-hlc8x" is forbidden: violates PodSecurity "restricted:latest": ...
The old Pod keeps running, new ones are rejected. All six items plus
the read-only root FS in one JSON patch (index 1 = volume
host):
k patch deploy backup --type=json -p='[
{"op": "remove", "path": "/spec/template/spec/volumes/1"},
{"op": "remove", "path": "/spec/template/spec/containers/0/volumeMounts/1"},
{"op": "add", "path": "/spec/template/spec/securityContext", "value":
{"runAsNonRoot": true, "runAsUser": 65532,
"seccompProfile": {"type": "RuntimeDefault"}}},
{"op": "replace", "path": "/spec/template/spec/containers/0/securityContext",
"value": {"allowPrivilegeEscalation": false, "readOnlyRootFilesystem": true,
"capabilities": {"drop": ["ALL"]}}}]'
k rollout status deploy/backup
k exec deploy/backup -- id
k exec deploy/backup -- touch /x
k exec deploy/backup -- sh -c 'date > /data/test && echo ok'
k label --dry-run=server --overwrite ns tag19-r1 \
pod-security.kubernetes.io/enforce=restrictedExpected (abridged):
deployment.apps/backup patched
deployment "backup" successfully rolled out
uid=65532 gid=0(root) groups=0(root)
touch: /x: Read-only file system
command terminated with exit code 1
ok
namespace/tag19-r1 not labeled (server dry run)
The server dry run without a warning proves it: no running Pod
violates restricted anymore. restricted does
not require readOnlyRootFilesystem – that was the extra
requirement.
Save your progress? Sign up for free
Round 2: The deployer is cluster admin
Situation: The pipeline uses the ServiceAccount
deployer in tag19-r2. It may read and change
Deployments in this namespace (get, list,
watch, update, patch) – nothing
else.
k create ns tag19-r2
k config set-context --current --namespace=tag19-r2
k create sa deployer
k create clusterrolebinding tag19-r2-deployer --clusterrole=cluster-admin \
--serviceaccount=tag19-r2:deployerTarget time: 3 min
Hint: The binding is not in the namespace –
k get rolebindings stays empty.
Solution
SA=system:serviceaccount:tag19-r2:deployer
k auth can-i get secrets -A --as=$SA
k get clusterrolebindings -o wide | grep -E 'NAME|deployer'Expected (abridged):
yes
NAME ROLE AGE ... SERVICEACCOUNTS
tag19-r2-deployer ClusterRole/cluster-admin 1s ... tag19-r2/deployer
Delete the binding, create a narrow Role:
k delete clusterrolebinding tag19-r2-deployer
k create role deployer --verb=get,list,watch,update,patch \
--resource=deployments.apps
k create rolebinding deployer --role=deployer --serviceaccount=tag19-r2:deployer
k auth can-i patch deployments -n tag19-r2 --as=$SA
k auth can-i create deployments -n tag19-r2 --as=$SA
k auth can-i get secrets -n tag19-r2 --as=$SA
k auth can-i patch deployments -n kube-system --as=$SAExpected:
clusterrolebinding.rbac.authorization.k8s.io "tag19-r2-deployer" deleted
role.rbac.authorization.k8s.io/deployer created
rolebinding.rbac.authorization.k8s.io/deployer created
yes
no
no
no
Four can-i prove both directions: what should be allowed
works; creating, Secrets and other namespaces do not.
Save your progress? Sign up for free
Round 3: The policy that allows everything
Situation: notes-api runs in tag19-r3.
Only Pods with the label role=frontend from the
same namespace may access port 8080; all other incoming traffic
into the namespace is to be blocked.
k create ns tag19-r3
k config set-context --current --namespace=tag19-r3
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 1
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: notes
spec:
selector:
app: notes
ports:
- port: 80
targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all
spec:
podSelector: {}
ingress:
- {}
EOF
k run frontend --image=busybox:1.37 -l role=frontend -- sleep 3600
k run fremd --image=busybox:1.37 -- sleep 3600
k create ns tag19-r3-gast
k -n tag19-r3-gast run gast --image=busybox:1.37 -l role=frontend -- sleep 3600Target time: 4 min
Hint: NetworkPolicies add up. A deny policy next to one that allows everything changes nothing.
Solution
Situation – all three get through:
k exec frontend -- wget -qO- -T3 notes/healthz
k exec fremd -- wget -qO- -T3 notes/healthz
k -n tag19-r3-gast exec gast -- wget -qO- -T3 notes.tag19-r3/healthzExpected:
ok
ok
ok
Default deny and the targeted allow:
k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: notes-from-frontend
spec:
podSelector:
matchLabels:
app: notes
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
role: frontend
ports:
- port: 8080
EOF
k exec fremd -- wget -qO- -T3 notes/healthz
k get netpolExpected:
networkpolicy.networking.k8s.io/default-deny created
networkpolicy.networking.k8s.io/notes-from-frontend created
ok
NAME POD-SELECTOR AGE
allow-all <none> 12s
default-deny <none> 0s
notes-from-frontend app=notes 0s
fremd still gets through. The reason is in
allow-all:
k describe netpol allow-all | sed -n '/Spec/,$p'
k delete netpol allow-all
k exec frontend -- wget -qO- -T3 notes/healthz
k exec fremd -- wget -qO- -T3 notes/healthz
k -n tag19-r3-gast exec gast -- wget -qO- -T3 notes.tag19-r3/healthzExpected:
Spec:
PodSelector: <none> (Allowing the specific traffic to all pods in this namespace)
Allowing ingress traffic:
To Port: <any> (traffic allowed to all ports)
From: <any> (traffic not restricted by source)
Not affecting egress traffic
Policy Types: Ingress
networkpolicy.networking.k8s.io "allow-all" deleted from tag19-r3 namespace
ok
wget: download timed out
command terminated with exit code 1
wget: download timed out
command terminated with exit code 1
gast does carry role=frontend, but it comes
from another namespace: a podSelector without a
namespaceSelector only applies within its own
namespace.
Save your progress? Sign up for free
Round 4: The open kubelet
Situation: “For debugging”, someone opened the
kubelet API on worker2. Find the hole from cp1
and close it. Setup from cp1 (backup first):
ssh [email protected]
sudo cp /var/lib/kubelet/config.yaml /root/kubelet-config.tag19
sudo sed -i -e '/anonymous:/{n;s/enabled: false/enabled: true/}' \
-e 's/ mode: Webhook/ mode: AlwaysAllow/' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
exitTarget time: 4 min
Hint: Test port 10250 on all nodes without
authentication. ps on the kubelet process reveals the
configuration file.
Solution
On cp1:
curl -sk https://172.16.0.228:10250/pods; echo
curl -sk https://172.16.0.83:10250/pods | jq -r '.items[].metadata.name' | head -2
P=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name | cut -d/ -f2)
curl -sk -X POST "https://172.16.0.83:10250/run/kube-system/$P/cilium-agent?cmd=id"Expected (abridged):
Unauthorized
cilium-zg9q2
falco-ld8gg
uid=0(root) gid=0(root) groups=0(root)
worker1 refuses, worker2 runs commands
anonymously as root. On worker2:
ssh [email protected]
ps -o args= -C kubelet | tr ' ' '\n' | grep -- --config
sudo grep -A1 -E " anonymous:|^authorization:" /var/lib/kubelet/config.yaml
sudo sed -i -e '/anonymous:/{n;s/enabled: true/enabled: false/}' \
-e 's/ mode: AlwaysAllow/ mode: Webhook/' /var/lib/kubelet/config.yaml
sudo grep -A1 -E " anonymous:|^authorization:" /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubelet
sudo cmp /var/lib/kubelet/config.yaml /root/kubelet-config.tag19 && echo gleich
sudo rm /root/kubelet-config.tag19
exitExpected:
--config=/var/lib/kubelet/config.yaml
anonymous:
enabled: true
--
authorization:
mode: AlwaysAllow
anonymous:
enabled: false
--
authorization:
mode: Webhook
active
gleich
Proof on cp1:
curl -sk https://172.16.0.83:10250/pods; echo
k get nodes worker2Expected (abridged):
Unauthorized
NAME STATUS ROLES AGE VERSION
worker2 Ready <none> 112m …
cmp against the backup shows: the file is bit-for-bit as
it was before the setup. The permissions 600 from Day 3
stay, because sed -i keeps them.
Save your progress? Sign up for free
Round 5: The node is allowed too much
Situation: An audit reports: a compromised worker
could give itself labels with the prefix
node-restriction.kubernetes.io/ and thus attract
security-relevant Pods. Find the cause on the API server and fix it.
Setup on cp1 (backup outside the manifest folder):
M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag19/backup/kube-apiserver.yaml
sudo sed -i '/--enable-admission-plugins=NodeRestriction/d' $M/kube-apiserver.yamlWait one to two minutes until k get --raw /readyz
reports ok again.
Target time: 5 min
Hint: Test with the kubelet’s kubeconfig on
worker1. Which CIS check covers admission plugins?
Solution
Proof on worker1 (from cp1 via
ssh [email protected]):
export KUBECONFIG=/etc/kubernetes/kubelet.conf
sudo -E kubectl auth whoami | grep Username
sudo -E kubectl label node worker1 node-restriction.kubernetes.io/tag19=pwn
exitExpected:
Username system:node:worker1
node/worker1 labeled
NodeRestriction would have forbidden that. On cp1:
sudo kube-bench run --targets master --benchmark cis-1.12 --check 1.2.14 \
| grep -E '^\[(FAIL|PASS)'
sudo grep -c admission $M/kube-apiserver.yamlExpected:
[FAIL] 1.2.14 Ensure that the admission control plugin NodeRestriction is set
(Automated)
0
Add the flag back – in the exam with vim, here via
sed directly after --client-ca-file:
sudo sed -i '/- --client-ca-file=/a\ - --enable-admission-plugins=NodeRestriction' \
$M/kube-apiserver.yaml
sudo grep -B1 -A1 -- '--enable-admission' $M/kube-apiserver.yaml
sudo md5sum $M/kube-apiserver.yaml ~/cks/tag19/backup/kube-apiserver.yamlExpected (checksums abridged):
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
… /etc/kubernetes/manifests/kube-apiserver.yaml
… /home/ubuntu/cks/tag19/backup/kube-apiserver.yaml
Both checksums are identical. After the restart (1–2 min):
k get --raw /readyz; echo
ps -o args= -C kube-apiserver | tr ' ' '\n' | grep admissionExpected:
ok
--enable-admission-plugins=NodeRestriction
Proof on worker1, then remove the smuggled-in label as
admin on cp1:
export KUBECONFIG=/etc/kubernetes/kubelet.conf
sudo -E kubectl label node worker1 node-restriction.kubernetes.io/tag19=pwn2 \
--overwrite
exitk label node worker1 node-restriction.kubernetes.io/tag19-Expected:
Error from server (Forbidden): nodes "worker1" is forbidden: is not allowed to modify
labels: node-restriction.kubernetes.io/tag19
node/worker1 unlabeled
kubeadm sets the flag on its own (built-in default, not visible in
kubeadm-config) – so an upgrade restores it. A label the
attacker has set, on the other hand, stays until you remove it.
Save your progress? Sign up for free
Round 6: Plaintext key, Secret with an onlooker
Situation: In tag19-r6, the notes-api
API key sits in plaintext in the Deployment. Put it into a Secret
notes-key. In addition, the audit log is to show who read
the Secret db-cred – you take away that account’s access to
Secrets (it may still read ConfigMaps).
k create ns tag19-r6
k config set-context --current --namespace=tag19-r6
k create deployment notes --image=172.16.0.126:30500/notes-api:1.0
k set env deploy/notes NOTES_API_KEY=lab-key-123
k create secret generic db-cred --from-literal=password=Lab-Geheim-19
k create sa reporter
k create role reporter --verb=get,list --resource=secrets,configmaps
k create rolebinding reporter --role=reporter --serviceaccount=tag19-r6:reporter
T=$(k create token reporter --duration=10m)
curl -s --cacert /etc/kubernetes/pki/ca.crt -H "Authorization: Bearer $T" \
https://172.16.0.126:6443/api/v1/namespaces/tag19-r6/secrets/db-cred > /dev/nullLab-Geheim-19 is a pure lab value.
Target time: 5 min
Hint: The audit log is in
/var/log/kubernetes/audit/audit.log (Day 15);
objectRef and user are the fields for
jq.
Solution
k get deploy notes -o jsonpath='{.spec.template.spec.containers[0].env}'; echo
L=/var/log/kubernetes/audit/audit.log
sudo jq -c 'select(.objectRef.resource=="secrets" and .objectRef.name=="db-cred"
and .objectRef.namespace=="tag19-r6" and .verb=="get")
| [.user.username, .verb, .responseStatus.code, .sourceIPs[0]]' $L
k get rolebindings -o wideExpected (abridged):
[{"name":"NOTES_API_KEY","value":"lab-key-123"}]
["system:serviceaccount:tag19-r6:reporter","get",200,"172.16.0.126"]
NAME ROLE AGE USERS GROUPS SERVICEACCOUNTS
reporter Role/reporter 7s tag19-r6/reporter
Key into the Secret, variable from the Secret, Role without Secrets:
k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123
k set env deploy/notes NOTES_API_KEY-
k set env deploy/notes --from=secret/notes-key
k rollout status deploy/notes
k set env deploy/notes --list
k create role reporter --verb=get,list --resource=configmaps $do | k apply -f -
SA=system:serviceaccount:tag19-r6:reporter
k auth can-i get secrets --as=$SA
k auth can-i list configmaps --as=$SA
k exec deploy/notes -- wget -qO- localhost:8080/healthz; echoExpected (abridged):
secret/notes-key created
deployment.apps/notes env updated
deployment.apps/notes env updated
deployment "notes" successfully rolled out
# Deployment notes, container notes-api
# NOTES_API_KEY from secret notes-key, key NOTES_API_KEY
Warning: resource roles/reporter is missing the kubectl.kubernetes.io/last-applied-...
role.rbac.authorization.k8s.io/reporter configured
no
yes
ok
The warning is harmless: apply adds the missing
annotation. Thanks to Day 10, the new Secret is stored encrypted in etcd
(k8s:enc:aescbc:v1:key2, see Checkpoint).
Save your progress? Sign up for free
Round 7: The old proxy image
Situation: No image with CRITICAL
vulnerabilities may run in tag19-r7. Replace affected
images with nginx:1.29-alpine-slim. Because the proxy
processes untrusted data, it should also run in the gVisor sandbox
(RuntimeClass gvisor, Day 11).
k create ns tag19-r7
k config set-context --current --namespace=tag19-r7
k create deployment proxy --image=nginx:1.19 --port=80
k expose deploy proxy --port=80
k create deployment notes --image=172.16.0.126:30500/notes-api:1.0
k set env deploy/notes NOTES_API_KEY=lab-key-123Target time: 5 min (Trivy takes a while per image)
Hint: First get the image list from the Pods, then
scan each image individually; --exit-code 1 saves you
reading the table.
Solution
k get pods -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' \
| sort -u
trivy image -q --severity CRITICAL nginx:1.19 | grep Total
trivy image -q --insecure --severity CRITICAL --exit-code 1 \
172.16.0.126:30500/notes-api:1.0 > /dev/null; echo $?
trivy image -q --severity CRITICAL --exit-code 1 nginx:1.29-alpine-slim \
> /dev/null; echo $?Expected:
172.16.0.126:30500/notes-api:1.0
nginx:1.19
Total: 42 (CRITICAL: 42)
0
0
Only the proxy is affected; the replacement image is clean (Trivy DB as of 2026-11-18). Swap the image and move it into the sandbox:
k set image deploy/proxy nginx=nginx:1.29-alpine-slim
k patch deploy proxy -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
k rollout status deploy/proxy
k get deploy -o custom-columns=NAME:.metadata.name,\
IMAGE:.spec.template.spec.containers[0].image,RC:.spec.template.spec.runtimeClassName
k exec deploy/proxy -- dmesg | head -1
k exec deploy/notes -- wget -qO- -S -T5 proxy 2>&1 | grep -E 'HTTP|Server'Expected (abridged):
deployment.apps/proxy image updated
deployment.apps/proxy patched
deployment "proxy" successfully rolled out
NAME IMAGE RC
notes 172.16.0.126:30500/notes-api:1.0 <none>
proxy nginx:1.29-alpine-slim gvisor
[ 0.000000] Starting gVisor...
HTTP/1.1 200 OK
Server: nginx/1.29.8
The container name nginx comes from
k create deployment (name of the image). The RuntimeClass
brings its own nodeSelector sandbox=gvisor –
the Pod only lands on prepared workers.
Save your progress? Sign up for free
Round 8: Falco raises the alarm
Situation: For minutes, Falco has been reporting
access to /etc/shadow. Find the Pod and its owner, cut the
Pod off from the network, save YAML and logs to
~/cks/tag19/evidence-r8/, remove it and make sure it does
not come back. web keeps running unchanged.
k create ns tag19-r8
k config set-context --current --namespace=tag19-r8
k create deployment web --image=nginx:1.29-alpine-slim --replicas=2
k create deployment sync --image=busybox:1.37 -- \
sh -c 'while true; do cat /etc/shadow > /dev/null; sleep 20; done'Wait a minute so Falco has something to report.
Target time: 5 min
Hint: Falco appends k8s_pod_name and
k8s_ns_name to every alert. You get the logs of all three
Falco Pods with a label selector.
Solution
k -n falco logs -l app.kubernetes.io/name=falco -c falco --since=5m --tail=-1 \
| grep 'Sensitive file' \
| grep -o 'file=[^ ]*\|k8s_pod_name=[^ ]*\|k8s_ns_name=[^ ]*' | sort | uniq -cExpected:
3 file=/etc/shadow
3 k8s_ns_name=tag19-r8
3 k8s_pod_name=sync-7cdd59b4db-gvfwc
Don’t get confused: alerts “Detected ptrace PTRACE_ATTACH attempt”
with containerd-shim as the parent process come from
starting the gVisor sandbox in round 7.
P=sync-7cdd59b4db-gvfwc
k get pod $P -o jsonpath='{.metadata.ownerReferences[0].name}{"\n"}'
k get pod $P -o jsonpath='{.spec.containers[0].command}{"\n"}'Expected:
sync-7cdd59b4db
["sh","-c","while true; do cat /etc/shadow > /dev/null; sleep 20; done"]
First the quarantine policy, then detach the Pod from the ReplicaSet, then shut down the owner:
k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: quarantine
spec:
podSelector:
matchLabels:
quarantine: "true"
policyTypes:
- Ingress
- Egress
EOF
k label pod $P app- quarantine=true
k scale deploy sync --replicas=0
k get pods --show-labels
k get cep $P -o jsonpath='{.status.identity.labels}' \
| grep -o 'k8s:quarantine=true'Empty? Wait briefly and repeat (the identity follows the labels with a delay, Day 18). Then:
k exec $P -- wget -qO- -T3 http://webExpected (abridged):
networkpolicy.networking.k8s.io/quarantine created
pod/sync-7cdd59b4db-gvfwc modified
deployment.apps/sync scaled
NAME READY STATUS ... LABELS
sync-7cdd59b4db-f7pjp 0/1 Terminating ... app=sync,pod-template-hash=...
sync-7cdd59b4db-gvfwc 1/1 Running ... ...,quarantine=true
web-846db866-q5dll 1/1 Running ... app=web,pod-template-hash=846db866
web-846db866-q99rb 1/1 Running ... app=web,pod-template-hash=846db866
wget: bad address 'web'
command terminated with exit code 1
The Terminating Pod is the replacement the ReplicaSet
started immediately after the relabeling – scale cleans it
up. bad address: even DNS is blocked for the quarantined
Pod. Secure the evidence, then delete:
mkdir -p ~/cks/tag19/evidence-r8
k get pod $P -o yaml > ~/cks/tag19/evidence-r8/pod.yaml
k logs $P > ~/cks/tag19/evidence-r8/log.txt
k delete pod $P --now
k get deployExpected (abridged):
pod "sync-7cdd59b4db-gvfwc" deleted from tag19-r8 namespace
NAME READY UP-TO-DATE AVAILABLE AGE
sync 0/0 0 0 71s
web 2/2 2 2 71s
One minute later:
k -n falco logs -l app.kubernetes.io/name=falco -c falco --since=30s --tail=-1 \
| grep -c tag19-r8Expected: 0
Stop the clock.
Save your progress? Sign up for free
Scoring
| Round | Hole | Target time | Points |
|---|---|---|---|
| 1 | privileged + hostPath, PSA restricted | 4 min | 2 |
| 2 | cluster-admin for ServiceAccount | 3 min | 2 |
| 3 | NetworkPolicy allow-all | 4 min | 2 |
| 4 | kubelet anonymous + AlwaysAllow | 4 min | 2 |
| 5 | NodeRestriction missing | 5 min | 2 |
| 6 | plaintext key + audit trail | 5 min | 3 |
| 7 | CRITICAL image + gVisor | 5 min | 2 |
| 8 | isolate the Falco finding | 5 min | 3 |
You only get full points within the target time, without a hint and with a proof command; otherwise half (round down). Whoever switches off a hardening gets 0 for the round. Overall target time: 35 min, maximum: 18 points.
| Points | Assessment |
|---|---|
| 15–18 | exam-ready – move on to the Practice Exam (Appendix B) |
| 10–14 | good; repeat the days of the rounds you missed |
| 0–9 | work through Days 5, 6, 9 and 15 once more |
Checkpoint
First the rounds, then the hardening that stays beyond this day.
k get deploy -A | grep tag19Expected: backup 1/1 (r1),
notes 1/1 (r3, r6, r7), proxy 1/1 (r7),
sync 0/0 and web 2/2 (r8).
k get ns -l pod-security.kubernetes.io/enforce=restrictedExpected: tag19-r1 Active …
k auth can-i get secrets -A --as=system:serviceaccount:tag19-r2:deployerExpected: no
curl -sk https://172.16.0.83:10250/pods; echoExpected: Unauthorized (likewise for
172.16.0.228)
ssh [email protected] 'sudo stat -c "%a %n" /var/lib/kubelet/config.yaml'Expected:
600 /var/lib/kubelet/config.yaml
ps -o args= -C kube-apiserver | tr ' ' '\n' \
| grep -E 'admission|profiling|authentication-config|encryption-pro|audit-log-pa'Expected:
--enable-admission-plugins=NodeRestriction
--profiling=false
--authentication-config=/etc/kubernetes/authn/authn-config.yaml
--encryption-provider-config=/etc/kubernetes/enc/enc.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
ps -o args= -C kube-scheduler | tr ' ' '\n' | grep profilingExpected: --profiling=false (likewise
for kube-controller-manager)
curl -sk https://172.16.0.126:6443/version | jq -r .codeExpected: 401
P=/etc/kubernetes/pki
E="--cacert=$P/etcd/ca.crt --cert=$P/apiserver-etcd-client.crt"
E="$E --key=$P/apiserver-etcd-client.key"
sudo etcdctl $E get /registry/secrets/tag19-r6/notes-key --print-value-only \
| hexdump -C | head -2Expected:
00000000 6b 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 3a 76 |k8s:enc:aescbc:v|
00000010 31 3a 6b 65 79 32 3a 66 bb 42 01 70 b6 e3 2b 65 |1:key2:f.B.p..+e|
k -n falco get ds falco
k get runtimeclass gvisorExpected: falco with 3
under DESIRED and READY; gvisor with handler
runsc.
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt statusExpected (abridged):
Encryption: Wireguard
Interface: cilium_wg0
Number of peers: 2
k get --raw='/readyz?verbose' | tail -1
k get nodes
k get pods -A --no-headers | grep -vc RunningExpected: readyz check passed, three
nodes Ready, 0.
Challenge
A colleague has hidden two leftovers in the cluster. Find them
cluster-wide, without knowing the namespace, and remove
them. Legitimate system Pods (Falco needs privileged)
stay.
k create ns tag19-ch
k -n tag19-ch create sa ci
k create clusterrolebinding tag19-ch-ci --clusterrole=cluster-admin \
--serviceaccount=tag19-ch:ci
k -n tag19-ch run debug --image=busybox:1.37 --privileged -- sleep 3600Hint: Two jq queries: Pods with
privileged or hostPath outside of
kube-system, and all bindings to
cluster-admin.
Solution
k get pods -A -o json | jq -r '.items[]
| select(.metadata.namespace != "kube-system")
| select(any(.spec.containers[]; .securityContext.privileged == true)
or any(.spec.volumes[]?; .hostPath))
| .metadata.namespace + "/" + .metadata.name'
k get clusterrolebindings -o json | jq -r '.items[]
| select(.roleRef.name == "cluster-admin")
| .metadata.name + " " + ([.subjects[]? | .kind + ":" + .name] | join(","))'Expected:
falco/falco-j78wr
falco/falco-ld8gg
falco/falco-vlb49
tag19-ch/debug
cluster-admin Group:system:masters
kubeadm:cluster-admins Group:kubeadm:cluster-admins
tag19-ch-ci ServiceAccount:ci
Falco and kubeadm’s two group bindings are intended;
debug and tag19-ch-ci are not:
k -n tag19-ch delete pod debug --now
k delete clusterrolebinding tag19-ch-ci
k auth can-i '*' '*' --as=system:serviceaccount:tag19-ch:ciExpected:
pod "debug" deleted from tag19-ch namespace
clusterrolebinding.rbac.authorization.k8s.io "tag19-ch-ci" deleted
no
Clean up (the cluster changes from rounds 4 and 5 have already been rolled back):
k config set-context --current --namespace=default
k delete ns tag19-r1 tag19-r2 tag19-r3 tag19-r3-gast tag19-r6 tag19-r7 tag19-r8 \
tag19-chSave your progress? Sign up for free
Exam Tip
- Finish every task with before/after proof (
auth can-i,wget,curlon 10250, server dry run): that way hidden second errors such asallow-all(round 3) or old Pods (round 1) stand out. - Before every change to a static Pod manifest or the kubelet
configuration: a copy outside of
/etc/kubernetes/manifests, afterwardsreadyzandk get nodes. An API server that doesn’t come back costs more points than the task is worth. - Never weaken the security requirement to get a Pod running: adapt the Pod to the requirement (round 1), not the requirement to the Pod.
Enterprise Note
- Policy as code prevents almost all of today’s holes before
deployment: PSA labels via namespace templates, Kyverno/Gatekeeper or
ValidatingAdmissionPolicy against
privileged,hostPath, foreign registries andcluster-adminbindings. - Node and API server configuration belongs in configuration management (Ansible, Cluster API, image pipelines) with drift detection; kube-bench as a Job.
- Image scans belong in CI and continuously in the cluster (e.g. Trivy Operator), because new CVEs make old images critical overnight.
- Falco alerts go to the SIEM and on-call via Falcosidekick; quarantine policy and evidence collection belong in the incident runbook, not in someone’s head.
- Regular “red team days” like this Competition show whether alerts, runbooks and responsibilities hold up when it counts.