Day 18: Immutable Containers and Forensics
Goal: You make containers immutable at runtime, find write traces and suspicious processes with built-in tools, and take a compromised Pod out of service. · Duration: approx. 75 min · Prerequisite: Day 9 and Day 2
Briefing
- CKS: Monitoring, Logging and Runtime Security – “Ensure immutability of containers at runtime” and “Investigate and identify phases of attack and bad actors within the environment”.
- Immutable means: read-only root filesystem, writable locations only as emptyDir, ideally no shell and no package manager in the image.
- Forensics happens on the node:
crictlgives you the container ID and host PID,/procgives you the process’s program, arguments, sockets and filesystem. - Falco (Day 16) and Tetragon (Day 17) report this kind of thing automatically – today only built-in tools, so you know what lies behind their alerts.
k get pod -o wide ──> Node ──> crictl ps ──> crictl inspect ──> PID
|
/proc/<pid>/{exe,cmdline,cgroup,root,mounts} · nsenter -n ss <─┘
mkdir -p ~/cks/tag18 && cd ~/cks/tag18
k create ns tag18
k config set-context --current --namespace=tag18
k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123(lab-key-123 is a pure lab value)
The image notes-api:1.0 must be in the registry
(otherwise build it as on Day 9).
Drill 1: notes-api with a read-only root FS
cat > notes.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: notes
labels:
app: notes
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
envFrom:
- secretRef:
name: notes-key
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
EOF
k apply -f notes.yaml
k wait --for=condition=Ready pod/notes
k exec notes -- wget -qO- localhost:8080/healthz
k exec notes -- touch /tmp/x
k exec notes -- awk '$2=="/"{print $1, $2, substr($4,1,2)}' /proc/mountsExpected:
pod/notes condition met
ok
touch: /tmp/x: Read-only file system
command terminated with exit code 1
overlay / ro
The overlay root FS is mounted ro. An attacker cannot
download tools and cannot swap out a binary. notes-api writes nothing to
disk, so it does not even need a /tmp.
Save your progress? Sign up for free
Drill 2: nginx needs writable locations
Most images write somewhere. The log tells you where:
k apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: web
spec:
containers:
- name: web
image: nginx:1.29
securityContext:
readOnlyRootFilesystem: true
EOFAfter about ten seconds:
k get pod web
k logs web | tail -1Expected:
NAME READY STATUS RESTARTS AGE
web 0/1 Error 1 (6s ago) 8s
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)
nginx writes cache and temp files under /var/cache/nginx
and the PID file to /run; /tmp is added as a
writable location for scripts. One emptyDir for each of these
directories:
k delete pod web --now
cat > web.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: web
labels:
run: web
spec:
containers:
- name: web
image: nginx:1.29
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- name: cache
mountPath: /var/cache/nginx
- name: run
mountPath: /run
- name: tmp
mountPath: /tmp
volumes:
- name: cache
emptyDir: {}
- name: run
emptyDir: {}
- name: tmp
emptyDir: {}
EOF
k apply -f web.yaml
k wait --for=condition=Ready pod/web
k exec web -- touch /usr/share/nginx/html/index.html
k exec web -- touch /tmp/ok && echo tmp okExpected:
pod/web condition met
touch: cannot touch '/usr/share/nginx/html/index.html': Read-only file system
command terminated with exit code 1
tmp ok
The web page can no longer be tampered with (no defacement),
/tmp stays usable. The log line
can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
is only a notice from the entrypoint script, not an error.
Save your progress? Sign up for free
Drill 3: Finding write traces on the node
A writable container in which “someone” drops a binary and appends to
/etc/passwd:
k run drift --image=busybox:1.37 -- sleep 1d
k wait --for=condition=Ready pod/drift
k exec drift -- sh -c 'cp /bin/busybox /usr/bin/kworker
echo x::0:0::/:/bin/sh >> /etc/passwd'
k get pod drift -o wideExpected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE ...
drift 1/1 Running 0 3s 10.244.1.177 worker2 ...
Every change lands in the writable overlay layer
(upperdir). On the node from the NODE column (here
worker2):
ssh [email protected]
ID=$(sudo crictl ps --namespace tag18 --name drift -q)
PID=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
U=$(sudo grep -o -m1 'upperdir=[^,]*' /proc/$PID/mounts | cut -d= -f2)
sudo find $U -type f -printf '%P\n'
sudo tail -1 $U/etc/passwd
exitExpected:
usr/bin/kworker
etc/hostname
etc/passwd
etc/hosts
etc/resolv.conf
x::0:0::/:/bin/sh
The container runtime creates hostname,
hosts and resolv.conf as mount points for the
kubelet’s files – normal. kworker and passwd
are the traces. With readOnlyRootFilesystem, only the three
mount points would be listed here.
Save your progress? Sign up for free
Drill 4: Tracking down shells and package managers
Whoever finds a shell can improvise; whoever finds apk
or apt-get can install more:
k exec notes -- sh -c 'for b in sh apk apt-get wget nc; do command -v $b; done'
k exec web -- sh -c 'for b in sh apk apt-get curl; do command -v $b; done'
k run noshell --image=registry.k8s.io/pause:3.10.2
k wait --for=condition=Ready pod/noshell
k exec noshell -- shExpected (abridged):
/bin/sh
/sbin/apk
/usr/bin/wget
/usr/bin/nc
/usr/bin/sh
/usr/bin/apt-get
/usr/bin/curl
error: Internal error occurred: ... exec: "sh": executable file not found in $PATH
notes-api on Alpine ships a shell, a package manager and
nc – the shrunk image from Day 13 doesn’t have them; here
we deliberately use 1.0. You examine an image without a
shell with an ephemeral container that shares the target’s process
namespace:
k debug -it noshell --image=busybox:1.37 --target=noshell -- psExpected (abridged):
PID USER TIME COMMAND
1 65535 0:00 /pause
18 root 0:00 ps
Which Pods are not immutable at all?
k get pods -o custom-columns='NAME:.metadata.name,IMAGE:.spec.containers[*].image,'\
'RO:.spec.containers[*].securityContext.readOnlyRootFilesystem'Expected:
NAME IMAGE RO
drift busybox:1.37 <none>
noshell registry.k8s.io/pause:3.10.2 <none>
notes 172.16.0.126:30500/notes-api:1.0 true
web nginx:1.29 true
Save your progress? Sign up for free
Drill 5: Examining a suspicious process
The Deployment shop appears to serve only a web page –
on the side, a bind shell runs on port 4444:
k create deploy shop --image=busybox:1.37 $do \
-- sh -c 'nc -lk -p 4444 -e /bin/sh & httpd -f -p 8080' > shop.yaml
k apply -f shop.yaml
k expose deploy shop --port=80 --target-port=8080
k run client --image=busybox:1.37 -- sleep 1d
k wait --for=condition=Ready pod/client
k get pod -l app=shop -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,\
IP:.status.podIP
IP=$(k get pod -l app=shop -o jsonpath='{.items[0].status.podIP}')
k exec client -- sh -c "echo id | nc -w 2 $IP 4444"Expected:
NAME NODE IP
shop-87c56f54f-5bk77 worker2 10.244.1.141
uid=0(root) gid=0(root) groups=0(root),10(wheel)
Whoever reaches port 4444 is root in the container.
k exec … ps would show the process – but in a compromised
container you don’t trust ps. So from the node in the NODE
column (here worker2):
ssh [email protected]
ID=$(sudo crictl ps --namespace tag18 --name busybox -q)
PID=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
echo $PID
ps -o pid,ppid,user,cmd --ppid $PID
NC=$(pgrep -x nc)
sudo ls -l /proc/$NC/exe
sudo cat /proc/$NC/cmdline | tr '\0' ' '; echo
sudo nsenter -t $PID -n ss -tlnpExpected:
13482
PID PPID USER CMD
13496 13482 root nc -lk -p 4444 -e /bin/sh
lrwxrwxrwx 1 root root 0 Nov 18 17:41 /proc/13496/exe -> /bin/nc
nc -lk -p 4444 -e /bin/sh
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 1 *:4444 *:* users:(("nc",pid=13496,fd=3))
LISTEN 0 9 *:8080 *:* users:(("httpd",pid=13482,…
crictl inspect gives you the host PID of the main
process (here httpd), ps --ppid its children.
exe shows the path inside the container,
/proc/<pid>/root/ opens its filesystem from the host.
ss on the host does not see port 4444 – it lives in the
Pod’s network namespace, hence nsenter -n.
Save your progress? Sign up for free
Drill 6: From the host process back to the Pod
Often it’s the other way round: an alert names only a PID on the node. The cgroup reveals the container:
cat /proc/$NC/cgroup
CID=$(grep -o 'cri-containerd-[0-9a-f]*' /proc/$NC/cgroup | cut -d- -f3)
sudo crictl ps --id $CID -o json | jq -r '.containers[0] | .metadata.name + " "
+ .labels["io.kubernetes.pod.name"] + " " + .labels["io.kubernetes.pod.namespace"]'
sudo crictl inspect $CID > ~/inspect-shop.json
exitExpected (path abridged):
0::/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode54eb65a_54ad_…
.slice/cri-containerd-62ab4e431ca47e04026def3c75802927dbf745fc5ed0d97dfb84264765afca5a
.scope
busybox shop-87c56f54f-5bk77 tag18
The cgroup contains the Pod UID and container ID; the kubelet sets
the labels io.kubernetes.pod.* on every container.
Container name, Pod and namespace – the report needs nothing more.
Save your progress? Sign up for free
Drill 7: Isolating the Pod
Don’t delete right away: first take it out of service, then secure
the evidence. If you remove the label app, the ReplicaSet
releases the Pod (it starts a replacement) and the Service no longer
sends traffic to it:
P=$(k get pod -l app=shop -o name)
k label $P app- quarantine=true
k get pods -l app=shop -L quarantine
k get pods -l quarantine=true
cat > quarantine.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: quarantine
spec:
podSelector:
matchLabels:
quarantine: "true"
policyTypes:
- Ingress
- Egress
EOF
k apply -f quarantine.yaml
k get cep ${P#pod/} -o jsonpath='{.status.identity.labels}' \
| grep -o 'k8s:quarantine=true'
k exec client -- sh -c "echo id | nc -w 2 $IP 4444"; echo "exit $?"Expected (abridged):
pod/shop-87c56f54f-5bk77 modified
NAME READY STATUS RESTARTS AGE QUARANTINE
shop-87c56f54f-fljs8 1/1 Running 0 2s
NAME READY STATUS RESTARTS AGE
shop-87c56f54f-5bk77 1/1 Running 0 23s
networkpolicy.networking.k8s.io/quarantine created
k8s:quarantine=true
command terminated with exit code 1
exit 1
A policy without rules but with both policyTypes blocks
everything in and out – the attacker’s back channel, too. Watch out with
Cilium: policies apply via the endpoint’s security identity, and it
follows the Pod labels with a delay (up to a minute in the lab). If the
grep line stays empty, the old identity still applies and
port 4444 is open – wait briefly and repeat, only then test.
Save your progress? Sign up for free
Drill 8: Securing evidence and removing the Pod
mkdir -p evidence
k get $P -o yaml > evidence/pod.yaml
k logs $P > evidence/pod.log
k get events --field-selector involvedObject.name=${P#pod/} > evidence/events.txt
ls evidence
k delete $P --now
k scale deploy shop --replicas=0
k get deploy shopExpected:
events.txt
pod.log
pod.yaml
pod "shop-87c56f54f-5bk77" deleted from tag18 namespace
deployment.apps/shop scaled
NAME READY UP-TO-DATE AVAILABLE AGE
shop 0/0 0 0 61s
The backdoor is in the Pod template – the replacement Pod had it,
too. That’s why you scale the Deployment to 0 (or delete it) until the
image is cleared up. The file also includes
~/inspect-shop.json on the node (Drill 6) – saved while the
container was still running.
Save your progress? Sign up for free
Exercise: Speed Drills
- Deployment
api(notes-api:1.0, Secretnotes-keyas env) with a read-only root FS and an emptyDir under/tmp. Target time: 3 min - List all Pods in
tag18in which a container does not havereadOnlyRootFilesystem: true. Target time: 1 min - Determine the host PID of the container in the Pod
notes. Target time: 2 min - Isolate the Pod
driftwith the existing policy. Target time: 20 s
Solution
k create deploy api --image=172.16.0.126:30500/notes-api:1.0 $do > api.yamlAdd to the Pod template:
spec:
containers:
- name: notes-api
image: 172.16.0.126:30500/notes-api:1.0
envFrom:
- secretRef:
name: notes-key
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}k apply -f api.yaml
k rollout status deploy/api
k exec deploy/api -- sh -c 'touch /tmp/ok && touch /x'
k get pods -o json | jq -r '.items[]
| select(any(.spec.containers[]; .securityContext.readOnlyRootFilesystem != true))
| .metadata.name'
k get pod notes -o jsonpath='{.spec.nodeName}{"\n"}'
ssh [email protected] # node from the line before, here worker2
POD=$(sudo crictl pods --namespace tag18 --name '^notes$' -q)
ID=$(sudo crictl ps --pod $POD -q)
sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID; echo
exit
k label pod drift quarantine=trueExpected (abridged):
touch: /x: Read-only file system
client
drift
noshell
worker2
13077
pod/drift labeled
Save your progress? Sign up for free
Checkpoint
k get pod web -o jsonpath='{.spec.containers[0].securityContext.readOnlyRootFilesystem}'Expected: true
k exec web -- touch /etc/xExpected:
touch: cannot touch '/etc/x': Read-only file system
k get netpol quarantine -o jsonpath='{.spec.policyTypes}{"\n"}'Expected: ["Ingress","Egress"]
k get deploy shopExpected: shop 0/0 0 0 …
Challenge
Apply the Deployment below. Afterwards, a process in a Pod listens on
TCP 31337. Find container, Pod and namespace using only tools on
the nodes, save the Pod YAML and the container log to
~/cks/tag18/evidence-cron/, isolate the Pod and make sure
the process does not come back.
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: cron
spec:
selector:
matchLabels:
app: cron
template:
metadata:
labels:
app: cron
spec:
containers:
- name: main
image: busybox:1.37
command: ["sleep", "infinity"]
- name: log
image: busybox:1.37
command: ["sh", "-c", "while true; do nc -l -p 31337 -e /bin/sh; done"]
EOFHint: ss -tlnp on the host does not see
the port, but pgrep sees every process on the node.
Solution
Check on both workers (172.16.0.228 and 172.16.0.83):
ssh [email protected]
sudo ss -tlnp | grep 31337
pgrep -a -x nc
P=$(pgrep -x nc)
sudo nsenter -t $P -n ss -tlnp
CID=$(grep -o 'cri-containerd-[0-9a-f]*' /proc/$P/cgroup | cut -d- -f3)
sudo crictl ps --id $CID -o json | jq -r '.containers[0] | .metadata.name + " "
+ .labels["io.kubernetes.pod.name"] + " " + .labels["io.kubernetes.pod.namespace"]'
exitExpected:
11099 nc -l -p 31337 -e /bin/sh
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 1 *:31337 *:* users:(("nc",pid=11099,fd=3))
log cron-66f4cf9fd7-flzwf tag18
The container log is the culprit, main is
unremarkable. Back on cp1:
P=<Pod name from the crictl output> # here cron-66f4cf9fd7-flzwf
k label pod $P app- quarantine=true
mkdir -p evidence-cron
k get pod $P -o yaml > evidence-cron/pod.yaml
k logs $P -c log > evidence-cron/log.txt
k scale deploy cron --replicas=0
k delete pod $P --now
k get deploy cron
k get pods -l quarantine=trueExpected (abridged):
pod/cron-66f4cf9fd7-flzwf modified
deployment.apps/cron scaled
pod "cron-66f4cf9fd7-flzwf" deleted from tag18 namespace
NAME READY UP-TO-DATE AVAILABLE AGE
cron 0/0 0 0 26s
NAME READY STATUS RESTARTS AGE
drift 1/1 Running 0 …
drift still carries the label from Speed Drill 4.
The relabeling has already started a replacement Pod with the same
backdoor – only scale --replicas=0 removes it. The detached
Pod no longer belongs to any ReplicaSet and can be deleted
afterwards.
Save your progress? Sign up for free
Cleanup
Applies to everyone, even without the challenge:
k config set-context --current --namespace=default
k delete ns tag18Exam Tip
- “Make the pod immutable” almost always means:
readOnlyRootFilesystem: trueplus emptyDir for the write paths from the log; often also removeprivilegedand enforce non-root. - “Delete pods that are not immutable / run as privileged”: search
with
custom-columnsorjq, check the result against the task, then fix the owner (Deployment), not just the Pod. crictlneedssudoon exam nodes;crictl inspectwithout a template is a long JSON –-o go-template --template '{{.info.pid}}'saves time.
Enterprise Note
- Distroless or
scratchimages (Day 13) take away the shell and package manager; debugging then runs viak debugwith ephemeral containers, granted narrowly via RBAC. - Enforce
readOnlyRootFilesystemvia admission (Kyverno, Gatekeeper or ValidatingAdmissionPolicy) – PSArestricteddoes not require it. - Detection should not be manual work: Falco (Day 16) reports shells, write access and netcat listeners, Tetragon (Day 17) can block them in the kernel.
- Define the incident runbook in advance: have a quarantine label and
a deny-all policy ready per namespace, secure evidence (YAML, logs,
crictl inspect, possibly a checkpoint) before deleting. - Treat the node itself as suspicious if the container had
privilegedorhostPID– cordon, drain, rebuild.