Skip to content
Hands-on IT IT Drills

Day 18: Immutable Containers and Forensics

Goal: You make containers immutable at runtime, find write traces and suspicious processes with built-in tools, and take a compromised Pod out of service. · Duration: approx. 75 min · Prerequisite: Day 9 and Day 2

Briefing

  • CKS: Monitoring, Logging and Runtime Security – “Ensure immutability of containers at runtime” and “Investigate and identify phases of attack and bad actors within the environment”.
  • Immutable means: read-only root filesystem, writable locations only as emptyDir, ideally no shell and no package manager in the image.
  • Forensics happens on the node: crictl gives you the container ID and host PID, /proc gives you the process’s program, arguments, sockets and filesystem.
  • Falco (Day 16) and Tetragon (Day 17) report this kind of thing automatically – today only built-in tools, so you know what lies behind their alerts.
k get pod -o wide ──> Node ──> crictl ps ──> crictl inspect ──> PID
                                                                 |
     /proc/<pid>/{exe,cmdline,cgroup,root,mounts} · nsenter -n ss <─┘
mkdir -p ~/cks/tag18 && cd ~/cks/tag18
k create ns tag18
k config set-context --current --namespace=tag18
k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123

(lab-key-123 is a pure lab value)

The image notes-api:1.0 must be in the registry (otherwise build it as on Day 9).

Drill 1: notes-api with a read-only root FS

cat > notes.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: notes
  labels:
    app: notes
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: notes-api
    image: 172.16.0.126:30500/notes-api:1.0
    envFrom:
    - secretRef:
        name: notes-key
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
EOF
k apply -f notes.yaml
k wait --for=condition=Ready pod/notes
k exec notes -- wget -qO- localhost:8080/healthz
k exec notes -- touch /tmp/x
k exec notes -- awk '$2=="/"{print $1, $2, substr($4,1,2)}' /proc/mounts

Expected:

pod/notes condition met
ok
touch: /tmp/x: Read-only file system
command terminated with exit code 1
overlay / ro

The overlay root FS is mounted ro. An attacker cannot download tools and cannot swap out a binary. notes-api writes nothing to disk, so it does not even need a /tmp.

Save your progress? Sign up for free

Drill 2: nginx needs writable locations

Most images write somewhere. The log tells you where:

k apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
  - name: web
    image: nginx:1.29
    securityContext:
      readOnlyRootFilesystem: true
EOF

After about ten seconds:

k get pod web
k logs web | tail -1

Expected:

NAME   READY   STATUS   RESTARTS     AGE
web    0/1     Error    1 (6s ago)   8s
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)

nginx writes cache and temp files under /var/cache/nginx and the PID file to /run; /tmp is added as a writable location for scripts. One emptyDir for each of these directories:

k delete pod web --now
cat > web.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: web
  labels:
    run: web
spec:
  containers:
  - name: web
    image: nginx:1.29
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: cache
      mountPath: /var/cache/nginx
    - name: run
      mountPath: /run
    - name: tmp
      mountPath: /tmp
  volumes:
  - name: cache
    emptyDir: {}
  - name: run
    emptyDir: {}
  - name: tmp
    emptyDir: {}
EOF
k apply -f web.yaml
k wait --for=condition=Ready pod/web
k exec web -- touch /usr/share/nginx/html/index.html
k exec web -- touch /tmp/ok && echo tmp ok

Expected:

pod/web condition met
touch: cannot touch '/usr/share/nginx/html/index.html': Read-only file system
command terminated with exit code 1
tmp ok

The web page can no longer be tampered with (no defacement), /tmp stays usable. The log line can not modify /etc/nginx/conf.d/default.conf (read-only file system?) is only a notice from the entrypoint script, not an error.

Save your progress? Sign up for free

Drill 3: Finding write traces on the node

A writable container in which “someone” drops a binary and appends to /etc/passwd:

k run drift --image=busybox:1.37 -- sleep 1d
k wait --for=condition=Ready pod/drift
k exec drift -- sh -c 'cp /bin/busybox /usr/bin/kworker
  echo x::0:0::/:/bin/sh >> /etc/passwd'
k get pod drift -o wide

Expected (abridged):

NAME    READY   STATUS    RESTARTS   AGE   IP             NODE      ...
drift   1/1     Running   0          3s    10.244.1.177   worker2   ...

Every change lands in the writable overlay layer (upperdir). On the node from the NODE column (here worker2):

ssh [email protected]
ID=$(sudo crictl ps --namespace tag18 --name drift -q)
PID=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
U=$(sudo grep -o -m1 'upperdir=[^,]*' /proc/$PID/mounts | cut -d= -f2)
sudo find $U -type f -printf '%P\n'
sudo tail -1 $U/etc/passwd
exit

Expected:

usr/bin/kworker
etc/hostname
etc/passwd
etc/hosts
etc/resolv.conf
x::0:0::/:/bin/sh

The container runtime creates hostname, hosts and resolv.conf as mount points for the kubelet’s files – normal. kworker and passwd are the traces. With readOnlyRootFilesystem, only the three mount points would be listed here.

Save your progress? Sign up for free

Drill 4: Tracking down shells and package managers

Whoever finds a shell can improvise; whoever finds apk or apt-get can install more:

k exec notes -- sh -c 'for b in sh apk apt-get wget nc; do command -v $b; done'
k exec web -- sh -c 'for b in sh apk apt-get curl; do command -v $b; done'
k run noshell --image=registry.k8s.io/pause:3.10.2
k wait --for=condition=Ready pod/noshell
k exec noshell -- sh

Expected (abridged):

/bin/sh
/sbin/apk
/usr/bin/wget
/usr/bin/nc
/usr/bin/sh
/usr/bin/apt-get
/usr/bin/curl
error: Internal error occurred: ... exec: "sh": executable file not found in $PATH

notes-api on Alpine ships a shell, a package manager and nc – the shrunk image from Day 13 doesn’t have them; here we deliberately use 1.0. You examine an image without a shell with an ephemeral container that shares the target’s process namespace:

k debug -it noshell --image=busybox:1.37 --target=noshell -- ps

Expected (abridged):

PID   USER     TIME  COMMAND
    1 65535     0:00 /pause
   18 root      0:00 ps

Which Pods are not immutable at all?

k get pods -o custom-columns='NAME:.metadata.name,IMAGE:.spec.containers[*].image,'\
'RO:.spec.containers[*].securityContext.readOnlyRootFilesystem'

Expected:

NAME      IMAGE                              RO
drift     busybox:1.37                       <none>
noshell   registry.k8s.io/pause:3.10.2       <none>
notes     172.16.0.126:30500/notes-api:1.0   true
web       nginx:1.29                         true

Save your progress? Sign up for free

Drill 5: Examining a suspicious process

The Deployment shop appears to serve only a web page – on the side, a bind shell runs on port 4444:

k create deploy shop --image=busybox:1.37 $do \
  -- sh -c 'nc -lk -p 4444 -e /bin/sh & httpd -f -p 8080' > shop.yaml
k apply -f shop.yaml
k expose deploy shop --port=80 --target-port=8080
k run client --image=busybox:1.37 -- sleep 1d
k wait --for=condition=Ready pod/client
k get pod -l app=shop -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,\
IP:.status.podIP
IP=$(k get pod -l app=shop -o jsonpath='{.items[0].status.podIP}')
k exec client -- sh -c "echo id | nc -w 2 $IP 4444"

Expected:

NAME                   NODE      IP
shop-87c56f54f-5bk77   worker2   10.244.1.141
uid=0(root) gid=0(root) groups=0(root),10(wheel)

Whoever reaches port 4444 is root in the container. k exec … ps would show the process – but in a compromised container you don’t trust ps. So from the node in the NODE column (here worker2):

ssh [email protected]
ID=$(sudo crictl ps --namespace tag18 --name busybox -q)
PID=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
echo $PID
ps -o pid,ppid,user,cmd --ppid $PID
NC=$(pgrep -x nc)
sudo ls -l /proc/$NC/exe
sudo cat /proc/$NC/cmdline | tr '\0' ' '; echo
sudo nsenter -t $PID -n ss -tlnp

Expected:

13482
    PID    PPID USER     CMD
  13496   13482 root     nc -lk -p 4444 -e /bin/sh
lrwxrwxrwx 1 root root 0 Nov 18 17:41 /proc/13496/exe -> /bin/nc
nc -lk -p 4444 -e /bin/sh
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      1                  *:4444            *:*    users:(("nc",pid=13496,fd=3))
LISTEN 0      9                  *:8080            *:*    users:(("httpd",pid=13482,…

crictl inspect gives you the host PID of the main process (here httpd), ps --ppid its children. exe shows the path inside the container, /proc/<pid>/root/ opens its filesystem from the host. ss on the host does not see port 4444 – it lives in the Pod’s network namespace, hence nsenter -n.

Save your progress? Sign up for free

Drill 6: From the host process back to the Pod

Often it’s the other way round: an alert names only a PID on the node. The cgroup reveals the container:

cat /proc/$NC/cgroup
CID=$(grep -o 'cri-containerd-[0-9a-f]*' /proc/$NC/cgroup | cut -d- -f3)
sudo crictl ps --id $CID -o json | jq -r '.containers[0] | .metadata.name + " "
  + .labels["io.kubernetes.pod.name"] + " " + .labels["io.kubernetes.pod.namespace"]'
sudo crictl inspect $CID > ~/inspect-shop.json
exit

Expected (path abridged):

0::/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode54eb65a_54ad_…
.slice/cri-containerd-62ab4e431ca47e04026def3c75802927dbf745fc5ed0d97dfb84264765afca5a
.scope
busybox shop-87c56f54f-5bk77 tag18

The cgroup contains the Pod UID and container ID; the kubelet sets the labels io.kubernetes.pod.* on every container. Container name, Pod and namespace – the report needs nothing more.

Save your progress? Sign up for free

Drill 7: Isolating the Pod

Don’t delete right away: first take it out of service, then secure the evidence. If you remove the label app, the ReplicaSet releases the Pod (it starts a replacement) and the Service no longer sends traffic to it:

P=$(k get pod -l app=shop -o name)
k label $P app- quarantine=true
k get pods -l app=shop -L quarantine
k get pods -l quarantine=true
cat > quarantine.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine
spec:
  podSelector:
    matchLabels:
      quarantine: "true"
  policyTypes:
  - Ingress
  - Egress
EOF
k apply -f quarantine.yaml
k get cep ${P#pod/} -o jsonpath='{.status.identity.labels}' \
  | grep -o 'k8s:quarantine=true'
k exec client -- sh -c "echo id | nc -w 2 $IP 4444"; echo "exit $?"

Expected (abridged):

pod/shop-87c56f54f-5bk77 modified
NAME                   READY   STATUS    RESTARTS   AGE   QUARANTINE
shop-87c56f54f-fljs8   1/1     Running   0          2s
NAME                   READY   STATUS    RESTARTS   AGE
shop-87c56f54f-5bk77   1/1     Running   0          23s
networkpolicy.networking.k8s.io/quarantine created
k8s:quarantine=true
command terminated with exit code 1
exit 1

A policy without rules but with both policyTypes blocks everything in and out – the attacker’s back channel, too. Watch out with Cilium: policies apply via the endpoint’s security identity, and it follows the Pod labels with a delay (up to a minute in the lab). If the grep line stays empty, the old identity still applies and port 4444 is open – wait briefly and repeat, only then test.

Save your progress? Sign up for free

Drill 8: Securing evidence and removing the Pod

mkdir -p evidence
k get $P -o yaml > evidence/pod.yaml
k logs $P > evidence/pod.log
k get events --field-selector involvedObject.name=${P#pod/} > evidence/events.txt
ls evidence
k delete $P --now
k scale deploy shop --replicas=0
k get deploy shop

Expected:

events.txt
pod.log
pod.yaml
pod "shop-87c56f54f-5bk77" deleted from tag18 namespace
deployment.apps/shop scaled
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
shop   0/0     0            0           61s

The backdoor is in the Pod template – the replacement Pod had it, too. That’s why you scale the Deployment to 0 (or delete it) until the image is cleared up. The file also includes ~/inspect-shop.json on the node (Drill 6) – saved while the container was still running.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Deployment api (notes-api:1.0, Secret notes-key as env) with a read-only root FS and an emptyDir under /tmp. Target time: 3 min
  2. List all Pods in tag18 in which a container does not have readOnlyRootFilesystem: true. Target time: 1 min
  3. Determine the host PID of the container in the Pod notes. Target time: 2 min
  4. Isolate the Pod drift with the existing policy. Target time: 20 s

Solution

k create deploy api --image=172.16.0.126:30500/notes-api:1.0 $do > api.yaml

Add to the Pod template:

    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        envFrom:
        - secretRef:
            name: notes-key
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: tmp
        emptyDir: {}
k apply -f api.yaml
k rollout status deploy/api
k exec deploy/api -- sh -c 'touch /tmp/ok && touch /x'
k get pods -o json | jq -r '.items[]
  | select(any(.spec.containers[]; .securityContext.readOnlyRootFilesystem != true))
  | .metadata.name'
k get pod notes -o jsonpath='{.spec.nodeName}{"\n"}'
ssh [email protected]         # node from the line before, here worker2
POD=$(sudo crictl pods --namespace tag18 --name '^notes$' -q)
ID=$(sudo crictl ps --pod $POD -q)
sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID; echo
exit
k label pod drift quarantine=true

Expected (abridged):

touch: /x: Read-only file system
client
drift
noshell
worker2
13077
pod/drift labeled

Save your progress? Sign up for free

Checkpoint

k get pod web -o jsonpath='{.spec.containers[0].securityContext.readOnlyRootFilesystem}'

Expected: true

k exec web -- touch /etc/x

Expected: touch: cannot touch '/etc/x': Read-only file system

k get netpol quarantine -o jsonpath='{.spec.policyTypes}{"\n"}'

Expected: ["Ingress","Egress"]

k get deploy shop

Expected: shop 0/0 0 0 …

Challenge

Apply the Deployment below. Afterwards, a process in a Pod listens on TCP 31337. Find container, Pod and namespace using only tools on the nodes, save the Pod YAML and the container log to ~/cks/tag18/evidence-cron/, isolate the Pod and make sure the process does not come back.

k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cron
spec:
  selector:
    matchLabels:
      app: cron
  template:
    metadata:
      labels:
        app: cron
    spec:
      containers:
      - name: main
        image: busybox:1.37
        command: ["sleep", "infinity"]
      - name: log
        image: busybox:1.37
        command: ["sh", "-c", "while true; do nc -l -p 31337 -e /bin/sh; done"]
EOF

Hint: ss -tlnp on the host does not see the port, but pgrep sees every process on the node.

Solution

Check on both workers (172.16.0.228 and 172.16.0.83):

ssh [email protected]
sudo ss -tlnp | grep 31337
pgrep -a -x nc
P=$(pgrep -x nc)
sudo nsenter -t $P -n ss -tlnp
CID=$(grep -o 'cri-containerd-[0-9a-f]*' /proc/$P/cgroup | cut -d- -f3)
sudo crictl ps --id $CID -o json | jq -r '.containers[0] | .metadata.name + " "
  + .labels["io.kubernetes.pod.name"] + " " + .labels["io.kubernetes.pod.namespace"]'
exit

Expected:

11099 nc -l -p 31337 -e /bin/sh
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN 0      1                  *:31337            *:*    users:(("nc",pid=11099,fd=3))
log cron-66f4cf9fd7-flzwf tag18

The container log is the culprit, main is unremarkable. Back on cp1:

P=<Pod name from the crictl output>   # here cron-66f4cf9fd7-flzwf
k label pod $P app- quarantine=true
mkdir -p evidence-cron
k get pod $P -o yaml > evidence-cron/pod.yaml
k logs $P -c log > evidence-cron/log.txt
k scale deploy cron --replicas=0
k delete pod $P --now
k get deploy cron
k get pods -l quarantine=true

Expected (abridged):

pod/cron-66f4cf9fd7-flzwf modified
deployment.apps/cron scaled
pod "cron-66f4cf9fd7-flzwf" deleted from tag18 namespace
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
cron   0/0     0            0           26s
NAME    READY   STATUS    RESTARTS   AGE
drift   1/1     Running   0          …

drift still carries the label from Speed Drill 4.

The relabeling has already started a replacement Pod with the same backdoor – only scale --replicas=0 removes it. The detached Pod no longer belongs to any ReplicaSet and can be deleted afterwards.

Save your progress? Sign up for free

Cleanup

Applies to everyone, even without the challenge:

k config set-context --current --namespace=default
k delete ns tag18

Exam Tip

  • “Make the pod immutable” almost always means: readOnlyRootFilesystem: true plus emptyDir for the write paths from the log; often also remove privileged and enforce non-root.
  • “Delete pods that are not immutable / run as privileged”: search with custom-columns or jq, check the result against the task, then fix the owner (Deployment), not just the Pod.
  • crictl needs sudo on exam nodes; crictl inspect without a template is a long JSON – -o go-template --template '{{.info.pid}}' saves time.

Enterprise Note

  • Distroless or scratch images (Day 13) take away the shell and package manager; debugging then runs via k debug with ephemeral containers, granted narrowly via RBAC.
  • Enforce readOnlyRootFilesystem via admission (Kyverno, Gatekeeper or ValidatingAdmissionPolicy) – PSA restricted does not require it.
  • Detection should not be manual work: Falco (Day 16) reports shells, write access and netcat listeners, Tetragon (Day 17) can block them in the kernel.
  • Define the incident runbook in advance: have a quarantine label and a deny-all policy ready per namespace, secure evidence (YAML, logs, crictl inspect, possibly a checkpoint) before deleting.
  • Treat the node itself as suspicious if the container had privileged or hostPID – cordon, drain, rebuild.