Day 8: AppArmor and seccomp
Goal: You load AppArmor and seccomp profiles on the workers, attach them to Pods via
securityContextand prove their effect inside the container and in the kernel log. · Duration: approx. 75 min · Prerequisite: Day 7
Briefing
- CKS: System Hardening – “Appropriately use kernel hardening tools such as AppArmor, seccomp”.
- AppArmor restricts files, network, capabilities via
a profile (name in the kernel), seccomp restricts
syscalls (system calls) via a JSON file under
/var/lib/kubelet/seccomp/. - Both must exist on every node the Pod can land on – Kubernetes does not distribute profiles.
- The field in both cases:
securityContext.<appArmorProfile|seccompProfile>withtype:RuntimeDefault,Localhost(pluslocalhostProfile) orUnconfined.
AppArmor: /etc/apparmor.d/<file> --apparmor_parser--> kernel
Pod: localhostProfile: <profile name>
seccomp: /var/lib/kubelet/seccomp/profiles/x.json (read by containerd)
Pod: localhostProfile: profiles/x.json
Preparation on cp1:
mkdir -p ~/cks/tag08 && cd ~/cks/tag08
k create ns tag08
k config set-context --current --namespace=tag08Drill 1: What a Pod gets without any settings
k run plain --image=busybox:1.37 -- sleep 1d
k wait --for=condition=Ready pod/plain
k exec plain -- cat /proc/1/attr/current; echo
k exec plain -- grep Seccomp: /proc/1/status
k exec plain -- unshare -U idExpected:
cri-containerd.apparmor.d (enforce)
Seccomp: 0
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody),65534(nobody)
AppArmor is already active: containerd sets up its default profile.
seccomp, on the other hand, is off (0) – the kubelet
default is Unconfined. That is why unshare -U
succeeds (a new user namespace, a popular escape helper). With the
runtime profile:
k run rd --image=busybox:1.37 --overrides='{"spec":{"securityContext":
{"seccompProfile":{"type":"RuntimeDefault"}}}}' -- sleep 1d
k wait --for=condition=Ready pod/rd
k exec rd -- grep Seccomp: /proc/1/status
k exec rd -- unshare -U idExpected:
Seccomp: 2
unshare: unshare(0x10000000): Operation not permitted
command terminated with exit code 1
2 means filter mode. RuntimeDefault blocks
around 50 rarely used, dangerous syscalls and breaks practically no
application – Day 9 makes it mandatory.
Save your progress? Sign up for free
Drill 2: Load an AppArmor profile
On worker1 (ssh [email protected]):
sudo tee /etc/apparmor.d/cks-deny-write <<'EOF'
#include <tunables/global>
profile cks-deny-write flags=(attach_disconnected) {
#include <abstractions/base>
file,
deny /** w,
}
EOF
sudo apparmor_parser -r /etc/apparmor.d/cks-deny-write
sudo aa-status | grep -E 'profiles are in enforce|cks'Expected:
25 profiles are in enforce mode.
cks-deny-write
The same on worker2
(ssh [email protected]). What matters is the name after
profile, not the file name. -r replaces a
loaded profile (the first time, -a is enough too),
-C loads it in complain mode (log only). Whatever lies in
/etc/apparmor.d/ is loaded again by the node at boot.
Save your progress? Sign up for free
Drill 3: Use the profile in a Pod
cat > aa.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: aa
spec:
containers:
- name: c
image: busybox:1.37
command: ["sleep", "1d"]
securityContext:
appArmorProfile:
type: Localhost
localhostProfile: cks-deny-write
EOF
k apply -f aa.yaml
k wait --for=condition=Ready pod/aa
k get pod aa -o wide | awk '{print $1, $3, $7}'
k exec aa -- cat /proc/1/attr/current; echo
k exec aa -- touch /tmp/x
k exec aa -- sh -c 'echo hi > /dev/null'Expected:
pod/aa created
pod/aa condition met
NAME STATUS NODE
aa Running worker2
cks-deny-write (enforce)
touch: /tmp/x: Permission denied
command terminated with exit code 1
sh: can't create /dev/null: Permission denied
command terminated with exit code 1
Even /dev/null is blocked – /** really
means everything. On the Pod’s node (here worker2),
sudo aa-status | grep cks now additionally shows
/usr/bin/sleep (22270) cks-deny-write.
Save your progress? Sign up for free
Drill 4: Violations in the kernel log
On the Pod’s node:
sudo journalctl -k --since -5min | grep 'apparmor="DENIED"'Expected: no output.
Explicit deny rules are silent. For forensics, put
audit in front; -r swaps the profile in the
kernel, even for the container that is already running. On
both workers:
sudo sed -i 's|^ deny /\*\* w,| audit deny /** w,|' /etc/apparmor.d/cks-deny-write
sudo apparmor_parser -r /etc/apparmor.d/cks-deny-writeOn cp1 run k exec aa -- touch /tmp/y, then
again on the Pod’s node:
sudo journalctl -k --since -2min | grep 'apparmor="DENIED"'Expected (shortened):
... worker2 kernel: audit: type=1400 audit(1795026631.899:204): apparmor="DENIED"
operation="mknod" class="file" profile="cks-deny-write" name="/tmp/y" pid=22715
comm="touch" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Save your progress? Sign up for free
Drill 5: Profile missing – and the old annotation
sed 's/name: aa/name: aa-miss/; s/cks-deny-write/cks-gibtsnicht/' aa.yaml \
| k apply -f -
k get pod aa-miss -w # Ctrl+C once the error status shows
k describe pod aa-miss | grep -m1 'Error:'
k delete pod aa-miss --nowExpected (shortened):
NAME READY STATUS RESTARTS AGE
aa-miss 0/1 CreateContainerError 0 15s
... Error: failed to get container spec opts: failed to generate apparmor spec opts:
apparmor profile not found cks-gibtsnicht
The Pod is accepted and scheduled; only containerd fails. In the exam
that means: load the profile on exactly the node the Pod runs on – or
send the Pod there via nodeName/nodeSelector.
Older tasks still use the annotation:
A=container.apparmor.security.beta.kubernetes.io/anno
k run anno --image=busybox:1.37 --annotations=$A=localhost/cks-deny-write -- sleep 1d
k get pod anno -o jsonpath='{.spec.containers[0].securityContext}{"\n"}'
k delete pod anno --nowExpected (shortened):
Warning: metadata.annotations[container.apparmor.security.beta.kubernetes.io/anno]:
deprecated since v1.30; use the "appArmorProfile" field instead
pod/anno created
{"appArmorProfile":{"localhostProfile":"cks-deny-write","type":"Localhost"}}
The API server translates the annotation
(localhost/<name>, runtime/default,
unconfined) into the field. New manifests use only the
field.
Save your progress? Sign up for free
Drill 6: A seccomp profile for recording
Profiles live relative to /var/lib/kubelet/seccomp/.
SCMP_ACT_LOG allows everything and logs every syscall. On
both workers:
sudo mkdir -p /var/lib/kubelet/seccomp/profiles
echo '{"defaultAction": "SCMP_ACT_LOG"}' \
| sudo tee /var/lib/kubelet/seccomp/profiles/audit.jsonOn cp1 – nodeName pins the node, so that
you know where the log is:
cat > audit.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: audit
spec:
nodeName: worker1
securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/audit.json
containers:
- name: c
image: busybox:1.37
command: ["sleep", "1d"]
EOF
k apply -f audit.yaml
k wait --for=condition=Ready pod/audit
k exec audit -- grep Seccomp: /proc/1/statusExpected: Seccomp: 2
Without auditd, the entries end up in the kernel log via
printk – throttled to 10 messages per 5 s. For the
experiment, switch that off briefly on worker1:
sudo sysctl -w kernel.printk_ratelimit=0On cp1 run k exec audit -- uname -r, then
on worker1:
sudo journalctl -k --since -1min | grep type=1326 | grep -o 'comm="[^"]*"' \
| sort | uniq -c
sudo journalctl -k --since -1min | grep type=1326 | grep 'comm="uname"' \
| grep -o 'syscall=[0-9]*' | sort -u | tr '\n' ' '; echo
sudo sysctl -w kernel.printk_ratelimit=5Expected (shortened):
532 comm="runc:[2:INIT]"
67 comm="uname"
syscall=0 syscall=1 syscall=10 syscall=102 syscall=12 syscall=158 ... syscall=63 ...
kernel.printk_ratelimit = 5
That is the raw list for a profile of your own: syscall 63 is
uname (x86_64; on arm64 it is 160). Tools such as the
Security Profiles Operator build an allow profile from it
automatically.
Save your progress? Sign up for free
Drill 7: Block a syscall
Allow everything except creating directories. On both workers:
sudo tee /var/lib/kubelet/seccomp/profiles/no-mkdir.json <<'EOF'
{
"defaultAction": "SCMP_ACT_ALLOW",
"syscalls": [
{
"names": ["mkdir", "mkdirat"],
"action": "SCMP_ACT_ERRNO"
}
]
}
EOFOn cp1:
k delete pod audit --now
sed 's/name: audit/name: nomkdir/; /nodeName/d; s#audit.json#no-mkdir.json#' \
audit.yaml > nomkdir.yaml
k apply -f nomkdir.yaml
k wait --for=condition=Ready pod/nomkdir
k exec nomkdir -- mkdir /tmp/neu
k exec nomkdir -- touch /tmp/datei
k exec nomkdir -- ls /tmpExpected:
mkdir: can't create directory '/tmp/neu': Operation not permitted
command terminated with exit code 1
datei
touch works (a different syscall), mkdir
does not. Both variants (mkdir, mkdirat)
belong in the list – which one a program uses depends on the libc.
Without the file:
sed 's/name: nomkdir/name: nofile/; s#no-mkdir.json#gibtsnicht.json#' nomkdir.yaml \
| k apply -f -
k get pod nofile -w # Ctrl+C once the error status shows
k describe pod nofile | grep -m1 'Error:'
k delete pod nofile --nowExpected (shortened):
... Error: failed to create containerd container: cannot load seccomp profile
"/var/lib/kubelet/seccomp/profiles/gibtsnicht.json": open
/var/lib/kubelet/seccomp/profiles/gibtsnicht.json: no such file or directory
Save your progress? Sign up for free
Drill 8: RuntimeDefault for everyone – seccompDefault
On Day 3, kube-bench reported 4.2.14
(--seccomp-default). The switch lives in the kubelet
configuration. On worker2, with a backup:
sudo cp -p /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.tag08
echo 'seccompDefault: true' | sudo tee -a /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubeletExpected:
seccompDefault: true
active
On cp1:
k run sd --image=busybox:1.37 --overrides='{"spec":{"nodeName":"worker2"}}' \
-- sleep 1d
k wait --for=condition=Ready pod/sd
k exec sd -- grep Seccomp: /proc/1/status
k get pod sd -o jsonpath='{.spec.securityContext}{"\n"}'Expected:
pod/sd condition met
Seccomp: 2
{}
Without a single word in the manifest, RuntimeDefault
applies. Running Pods keep their old profile until they restart. In the
lab, revert it (on worker2) so that later days find the
initial state:
sudo mv /var/lib/kubelet/config.yaml.tag08 /var/lib/kubelet/config.yaml
sudo systemctl restart kubeletSave your progress? Sign up for free
Exercise: Speed Drills
- Check whether the AppArmor profile
cks-deny-writeis loaded onworker1. Target time: 30 s - Pod
rd2(busybox:1.37,sleep 1d) with seccompRuntimeDefaultfor the whole Pod. Target time: 1 min - List all Pods in the cluster that use a
LocalhostAppArmor profile (Pod or container level). Target time: 2 min - Pod
nomkdir2withprofiles/no-mkdir.jsononly for the containerc. Target time: 1 min
Solution
ssh [email protected] 'sudo aa-status | grep cks-deny-write'
k run rd2 --image=busybox:1.37 $do -- sleep 1d > rd2.yamlIn rd2.yaml, add under spec: and apply:
securityContext:
seccompProfile:
type: RuntimeDefaultk apply -f rd2.yaml
k get pods -A -o json | jq -r '.items[]
| select([.spec.securityContext.appArmorProfile.type,
.spec.containers[].securityContext.appArmorProfile.type]
| index("Localhost"))
| .metadata.namespace + "/" + .metadata.name'For task 4, create a copy of nomkdir.yaml as
nomkdir2.yaml, change the name and move the block under the
container:
containers:
- name: c
image: busybox:1.37
command: ["sleep", "1d"]
securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/no-mkdir.jsonk apply -f nomkdir2.yaml
k exec nomkdir2 -- mkdir /tmp/aExpected (shortened):
cks-deny-write
pod/rd2 created
tag08/aa
...
pod/nomkdir2 created
mkdir: can't create directory '/tmp/a': Operation not permitted
Save your progress? Sign up for free
Checkpoint
ssh [email protected] 'sudo aa-status | grep cks-'
ssh [email protected] 'sudo aa-status | grep cks-'Expected: cks-deny-etc and
cks-deny-write on each (after the Challenge).
ssh [email protected] 'ls /var/lib/kubelet/seccomp/profiles'
ssh [email protected] 'ls /var/lib/kubelet/seccomp/profiles'Expected: audit.json no-mkdir.json on
each
ssh [email protected] 'sudo grep -c seccompDefault /var/lib/kubelet/config.yaml'Expected: 0
k get ns tag08Expected (after cleaning up):
Error from server (NotFound): namespaces "tag08" not found
What stays in the lab: the AppArmor profiles
cks-deny-write and cks-deny-etc as well as the
seccomp profiles audit.json and no-mkdir.json
on both workers. They do not get in the way as long as no Pod requests
them.
Challenge
Deployment etc (2 replicas, busybox:1.37,
sleep 1d) may write to /tmp, but nothing under
/etc – no matter which worker the Pods land on. Violations
should appear in the kernel log.
Hint: A profile at Pod level applies to all containers; think of the second worker.
Solution
On both workers:
sudo tee /etc/apparmor.d/cks-deny-etc <<'EOF'
#include <tunables/global>
profile cks-deny-etc flags=(attach_disconnected) {
#include <abstractions/base>
file,
audit deny /etc/** w,
}
EOF
sudo apparmor_parser -r /etc/apparmor.d/cks-deny-etcOn cp1:
k create deploy etc --image=busybox:1.37 --replicas=2 $do -- sleep 1d > etc.yamlIn etc.yaml, add under template.spec::
securityContext:
appArmorProfile:
type: Localhost
localhostProfile: cks-deny-etck apply -f etc.yaml
k rollout status deploy/etc
k get pods -l app=etc -o wide | awk '{print $1, $3, $7}'
for p in $(k get pods -l app=etc -o name); do
k exec $p -- sh -c 'touch /tmp/ok && echo tmp-ok; touch /etc/neu'
doneExpected (shortened):
deployment "etc" successfully rolled out
NAME STATUS NODE
etc-844bd7b7ff-kjzvs Running worker1
etc-844bd7b7ff-t5bp2 Running worker2
tmp-ok
touch: /etc/neu: Permission denied
tmp-ok
touch: /etc/neu: Permission denied
Clean up:
k config set-context --current --namespace=default
k delete ns tag08Save your progress? Sign up for free
Exam Tip
- AppArmor:
localhostProfileis the profile name fromaa-status, not the file path. seccomp: the path relative to/var/lib/kubelet/seccomp/(profiles/x.json). CreateContainerErrorafter creation almost always means: the profile is missing on the node, or there is a typo.k describe podnames it,k get pod -o widethe node.- Docs: kubernetes.io → Tutorials → Security → “Restrict a Container’s Access to Resources with AppArmor” and “Restrict a Container’s Syscalls with seccomp”.
Enterprise Note
- The Security Profiles Operator distributes profiles (CRDs
AppArmorProfile,SeccompProfile, recording viaProfileRecording) instead of manual work via ssh. seccompDefault: truein the kubelet configuration of all nodes is a cheap win; test workloads withRuntimeDefaultfirst.- Custom allow profiles only for exposed, stable services – every new library version can need new syscalls.
- The RHEL family uses SELinux instead of AppArmor
(
seLinuxOptions); the principle stays the same.