Skip to content
Hands-on IT IT Drills

Day 8: AppArmor and seccomp

Goal: You load AppArmor and seccomp profiles on the workers, attach them to Pods via securityContext and prove their effect inside the container and in the kernel log. · Duration: approx. 75 min · Prerequisite: Day 7

Briefing

  • CKS: System Hardening – “Appropriately use kernel hardening tools such as AppArmor, seccomp”.
  • AppArmor restricts files, network, capabilities via a profile (name in the kernel), seccomp restricts syscalls (system calls) via a JSON file under /var/lib/kubelet/seccomp/.
  • Both must exist on every node the Pod can land on – Kubernetes does not distribute profiles.
  • The field in both cases: securityContext.<appArmorProfile|seccompProfile> with type: RuntimeDefault, Localhost (plus localhostProfile) or Unconfined.
AppArmor: /etc/apparmor.d/<file> --apparmor_parser--> kernel
          Pod: localhostProfile: <profile name>
seccomp:  /var/lib/kubelet/seccomp/profiles/x.json (read by containerd)
          Pod: localhostProfile: profiles/x.json

Preparation on cp1:

mkdir -p ~/cks/tag08 && cd ~/cks/tag08
k create ns tag08
k config set-context --current --namespace=tag08

Drill 1: What a Pod gets without any settings

k run plain --image=busybox:1.37 -- sleep 1d
k wait --for=condition=Ready pod/plain
k exec plain -- cat /proc/1/attr/current; echo
k exec plain -- grep Seccomp: /proc/1/status
k exec plain -- unshare -U id

Expected:

cri-containerd.apparmor.d (enforce)
Seccomp:    0
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody),65534(nobody)

AppArmor is already active: containerd sets up its default profile. seccomp, on the other hand, is off (0) – the kubelet default is Unconfined. That is why unshare -U succeeds (a new user namespace, a popular escape helper). With the runtime profile:

k run rd --image=busybox:1.37 --overrides='{"spec":{"securityContext":
  {"seccompProfile":{"type":"RuntimeDefault"}}}}' -- sleep 1d
k wait --for=condition=Ready pod/rd
k exec rd -- grep Seccomp: /proc/1/status
k exec rd -- unshare -U id

Expected:

Seccomp:    2
unshare: unshare(0x10000000): Operation not permitted
command terminated with exit code 1

2 means filter mode. RuntimeDefault blocks around 50 rarely used, dangerous syscalls and breaks practically no application – Day 9 makes it mandatory.

Save your progress? Sign up for free

Drill 2: Load an AppArmor profile

On worker1 (ssh [email protected]):

sudo tee /etc/apparmor.d/cks-deny-write <<'EOF'
#include <tunables/global>

profile cks-deny-write flags=(attach_disconnected) {
  #include <abstractions/base>

  file,

  deny /** w,
}
EOF
sudo apparmor_parser -r /etc/apparmor.d/cks-deny-write
sudo aa-status | grep -E 'profiles are in enforce|cks'

Expected:

25 profiles are in enforce mode.
   cks-deny-write

The same on worker2 (ssh [email protected]). What matters is the name after profile, not the file name. -r replaces a loaded profile (the first time, -a is enough too), -C loads it in complain mode (log only). Whatever lies in /etc/apparmor.d/ is loaded again by the node at boot.

Save your progress? Sign up for free

Drill 3: Use the profile in a Pod

cat > aa.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: aa
spec:
  containers:
  - name: c
    image: busybox:1.37
    command: ["sleep", "1d"]
    securityContext:
      appArmorProfile:
        type: Localhost
        localhostProfile: cks-deny-write
EOF
k apply -f aa.yaml
k wait --for=condition=Ready pod/aa
k get pod aa -o wide | awk '{print $1, $3, $7}'
k exec aa -- cat /proc/1/attr/current; echo
k exec aa -- touch /tmp/x
k exec aa -- sh -c 'echo hi > /dev/null'

Expected:

pod/aa created
pod/aa condition met
NAME STATUS NODE
aa Running worker2
cks-deny-write (enforce)
touch: /tmp/x: Permission denied
command terminated with exit code 1
sh: can't create /dev/null: Permission denied
command terminated with exit code 1

Even /dev/null is blocked – /** really means everything. On the Pod’s node (here worker2), sudo aa-status | grep cks now additionally shows /usr/bin/sleep (22270) cks-deny-write.

Save your progress? Sign up for free

Drill 4: Violations in the kernel log

On the Pod’s node:

sudo journalctl -k --since -5min | grep 'apparmor="DENIED"'

Expected: no output.

Explicit deny rules are silent. For forensics, put audit in front; -r swaps the profile in the kernel, even for the container that is already running. On both workers:

sudo sed -i 's|^  deny /\*\* w,|  audit deny /** w,|' /etc/apparmor.d/cks-deny-write
sudo apparmor_parser -r /etc/apparmor.d/cks-deny-write

On cp1 run k exec aa -- touch /tmp/y, then again on the Pod’s node:

sudo journalctl -k --since -2min | grep 'apparmor="DENIED"'

Expected (shortened):

... worker2 kernel: audit: type=1400 audit(1795026631.899:204): apparmor="DENIED"
operation="mknod" class="file" profile="cks-deny-write" name="/tmp/y" pid=22715
comm="touch" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

Save your progress? Sign up for free

Drill 5: Profile missing – and the old annotation

sed 's/name: aa/name: aa-miss/; s/cks-deny-write/cks-gibtsnicht/' aa.yaml \
  | k apply -f -
k get pod aa-miss -w          # Ctrl+C once the error status shows
k describe pod aa-miss | grep -m1 'Error:'
k delete pod aa-miss --now

Expected (shortened):

NAME      READY   STATUS                 RESTARTS   AGE
aa-miss   0/1     CreateContainerError   0          15s
  ... Error: failed to get container spec opts: failed to generate apparmor spec opts:
apparmor profile not found cks-gibtsnicht

The Pod is accepted and scheduled; only containerd fails. In the exam that means: load the profile on exactly the node the Pod runs on – or send the Pod there via nodeName/nodeSelector. Older tasks still use the annotation:

A=container.apparmor.security.beta.kubernetes.io/anno
k run anno --image=busybox:1.37 --annotations=$A=localhost/cks-deny-write -- sleep 1d
k get pod anno -o jsonpath='{.spec.containers[0].securityContext}{"\n"}'
k delete pod anno --now

Expected (shortened):

Warning: metadata.annotations[container.apparmor.security.beta.kubernetes.io/anno]:
deprecated since v1.30; use the "appArmorProfile" field instead
pod/anno created
{"appArmorProfile":{"localhostProfile":"cks-deny-write","type":"Localhost"}}

The API server translates the annotation (localhost/<name>, runtime/default, unconfined) into the field. New manifests use only the field.

Save your progress? Sign up for free

Drill 6: A seccomp profile for recording

Profiles live relative to /var/lib/kubelet/seccomp/. SCMP_ACT_LOG allows everything and logs every syscall. On both workers:

sudo mkdir -p /var/lib/kubelet/seccomp/profiles
echo '{"defaultAction": "SCMP_ACT_LOG"}' \
  | sudo tee /var/lib/kubelet/seccomp/profiles/audit.json

On cp1 – nodeName pins the node, so that you know where the log is:

cat > audit.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: audit
spec:
  nodeName: worker1
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/audit.json
  containers:
  - name: c
    image: busybox:1.37
    command: ["sleep", "1d"]
EOF
k apply -f audit.yaml
k wait --for=condition=Ready pod/audit
k exec audit -- grep Seccomp: /proc/1/status

Expected: Seccomp: 2

Without auditd, the entries end up in the kernel log via printk – throttled to 10 messages per 5 s. For the experiment, switch that off briefly on worker1:

sudo sysctl -w kernel.printk_ratelimit=0

On cp1 run k exec audit -- uname -r, then on worker1:

sudo journalctl -k --since -1min | grep type=1326 | grep -o 'comm="[^"]*"' \
  | sort | uniq -c
sudo journalctl -k --since -1min | grep type=1326 | grep 'comm="uname"' \
  | grep -o 'syscall=[0-9]*' | sort -u | tr '\n' ' '; echo
sudo sysctl -w kernel.printk_ratelimit=5

Expected (shortened):

    532 comm="runc:[2:INIT]"
     67 comm="uname"
syscall=0 syscall=1 syscall=10 syscall=102 syscall=12 syscall=158 ... syscall=63 ...
kernel.printk_ratelimit = 5

That is the raw list for a profile of your own: syscall 63 is uname (x86_64; on arm64 it is 160). Tools such as the Security Profiles Operator build an allow profile from it automatically.

Save your progress? Sign up for free

Drill 7: Block a syscall

Allow everything except creating directories. On both workers:

sudo tee /var/lib/kubelet/seccomp/profiles/no-mkdir.json <<'EOF'
{
  "defaultAction": "SCMP_ACT_ALLOW",
  "syscalls": [
    {
      "names": ["mkdir", "mkdirat"],
      "action": "SCMP_ACT_ERRNO"
    }
  ]
}
EOF

On cp1:

k delete pod audit --now
sed 's/name: audit/name: nomkdir/; /nodeName/d; s#audit.json#no-mkdir.json#' \
  audit.yaml > nomkdir.yaml
k apply -f nomkdir.yaml
k wait --for=condition=Ready pod/nomkdir
k exec nomkdir -- mkdir /tmp/neu
k exec nomkdir -- touch /tmp/datei
k exec nomkdir -- ls /tmp

Expected:

mkdir: can't create directory '/tmp/neu': Operation not permitted
command terminated with exit code 1
datei

touch works (a different syscall), mkdir does not. Both variants (mkdir, mkdirat) belong in the list – which one a program uses depends on the libc. Without the file:

sed 's/name: nomkdir/name: nofile/; s#no-mkdir.json#gibtsnicht.json#' nomkdir.yaml \
  | k apply -f -
k get pod nofile -w           # Ctrl+C once the error status shows
k describe pod nofile | grep -m1 'Error:'
k delete pod nofile --now

Expected (shortened):

  ... Error: failed to create containerd container: cannot load seccomp profile
"/var/lib/kubelet/seccomp/profiles/gibtsnicht.json": open
/var/lib/kubelet/seccomp/profiles/gibtsnicht.json: no such file or directory

Save your progress? Sign up for free

Drill 8: RuntimeDefault for everyone – seccompDefault

On Day 3, kube-bench reported 4.2.14 (--seccomp-default). The switch lives in the kubelet configuration. On worker2, with a backup:

sudo cp -p /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.tag08
echo 'seccompDefault: true' | sudo tee -a /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubelet

Expected:

seccompDefault: true
active

On cp1:

k run sd --image=busybox:1.37 --overrides='{"spec":{"nodeName":"worker2"}}' \
  -- sleep 1d
k wait --for=condition=Ready pod/sd
k exec sd -- grep Seccomp: /proc/1/status
k get pod sd -o jsonpath='{.spec.securityContext}{"\n"}'

Expected:

pod/sd condition met
Seccomp:    2
{}

Without a single word in the manifest, RuntimeDefault applies. Running Pods keep their old profile until they restart. In the lab, revert it (on worker2) so that later days find the initial state:

sudo mv /var/lib/kubelet/config.yaml.tag08 /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Check whether the AppArmor profile cks-deny-write is loaded on worker1. Target time: 30 s
  2. Pod rd2 (busybox:1.37, sleep 1d) with seccomp RuntimeDefault for the whole Pod. Target time: 1 min
  3. List all Pods in the cluster that use a Localhost AppArmor profile (Pod or container level). Target time: 2 min
  4. Pod nomkdir2 with profiles/no-mkdir.json only for the container c. Target time: 1 min

Solution

ssh [email protected] 'sudo aa-status | grep cks-deny-write'
k run rd2 --image=busybox:1.37 $do -- sleep 1d > rd2.yaml

In rd2.yaml, add under spec: and apply:

  securityContext:
    seccompProfile:
      type: RuntimeDefault
k apply -f rd2.yaml
k get pods -A -o json | jq -r '.items[]
  | select([.spec.securityContext.appArmorProfile.type,
            .spec.containers[].securityContext.appArmorProfile.type]
           | index("Localhost"))
  | .metadata.namespace + "/" + .metadata.name'

For task 4, create a copy of nomkdir.yaml as nomkdir2.yaml, change the name and move the block under the container:

  containers:
  - name: c
    image: busybox:1.37
    command: ["sleep", "1d"]
    securityContext:
      seccompProfile:
        type: Localhost
        localhostProfile: profiles/no-mkdir.json
k apply -f nomkdir2.yaml
k exec nomkdir2 -- mkdir /tmp/a

Expected (shortened):

   cks-deny-write
pod/rd2 created
tag08/aa
...
pod/nomkdir2 created
mkdir: can't create directory '/tmp/a': Operation not permitted

Save your progress? Sign up for free

Checkpoint

ssh [email protected] 'sudo aa-status | grep cks-'
ssh [email protected] 'sudo aa-status | grep cks-'

Expected: cks-deny-etc and cks-deny-write on each (after the Challenge).

ssh [email protected] 'ls /var/lib/kubelet/seccomp/profiles'
ssh [email protected] 'ls /var/lib/kubelet/seccomp/profiles'

Expected: audit.json no-mkdir.json on each

ssh [email protected] 'sudo grep -c seccompDefault /var/lib/kubelet/config.yaml'

Expected: 0

k get ns tag08

Expected (after cleaning up): Error from server (NotFound): namespaces "tag08" not found

What stays in the lab: the AppArmor profiles cks-deny-write and cks-deny-etc as well as the seccomp profiles audit.json and no-mkdir.json on both workers. They do not get in the way as long as no Pod requests them.

Challenge

Deployment etc (2 replicas, busybox:1.37, sleep 1d) may write to /tmp, but nothing under /etc – no matter which worker the Pods land on. Violations should appear in the kernel log.

Hint: A profile at Pod level applies to all containers; think of the second worker.

Solution

On both workers:

sudo tee /etc/apparmor.d/cks-deny-etc <<'EOF'
#include <tunables/global>

profile cks-deny-etc flags=(attach_disconnected) {
  #include <abstractions/base>

  file,

  audit deny /etc/** w,
}
EOF
sudo apparmor_parser -r /etc/apparmor.d/cks-deny-etc

On cp1:

k create deploy etc --image=busybox:1.37 --replicas=2 $do -- sleep 1d > etc.yaml

In etc.yaml, add under template.spec::

      securityContext:
        appArmorProfile:
          type: Localhost
          localhostProfile: cks-deny-etc
k apply -f etc.yaml
k rollout status deploy/etc
k get pods -l app=etc -o wide | awk '{print $1, $3, $7}'
for p in $(k get pods -l app=etc -o name); do
  k exec $p -- sh -c 'touch /tmp/ok && echo tmp-ok; touch /etc/neu'
done

Expected (shortened):

deployment "etc" successfully rolled out
NAME STATUS NODE
etc-844bd7b7ff-kjzvs Running worker1
etc-844bd7b7ff-t5bp2 Running worker2
tmp-ok
touch: /etc/neu: Permission denied
tmp-ok
touch: /etc/neu: Permission denied

Clean up:

k config set-context --current --namespace=default
k delete ns tag08

Save your progress? Sign up for free

Exam Tip

  • AppArmor: localhostProfile is the profile name from aa-status, not the file path. seccomp: the path relative to /var/lib/kubelet/seccomp/ (profiles/x.json).
  • CreateContainerError after creation almost always means: the profile is missing on the node, or there is a typo. k describe pod names it, k get pod -o wide the node.
  • Docs: kubernetes.io → Tutorials → Security → “Restrict a Container’s Access to Resources with AppArmor” and “Restrict a Container’s Syscalls with seccomp”.

Enterprise Note

  • The Security Profiles Operator distributes profiles (CRDs AppArmorProfile, SeccompProfile, recording via ProfileRecording) instead of manual work via ssh.
  • seccompDefault: true in the kubelet configuration of all nodes is a cheap win; test workloads with RuntimeDefault first.
  • Custom allow profiles only for exposed, stable services – every new library version can need new syscalls.
  • The RHEL family uses SELinux instead of AppArmor (seLinuxOptions); the principle stays the same.