Day 8: Security in the Pod
Goal: All notes Pods run under Pod Security “restricted”: without root, without capabilities, with a read-only file system and without a token for the Kubernetes API. · Duration: approx. 90 min · Prerequisite: Day 7
Briefing
- So far, whatever the image and runtime specify applies:
capabilities, a writable file system and the token of the ServiceAccount
defaultin every container. - Pod Security Admission checks every new Pod against one of three
levels:
privileged,baseline,restricted. Labels on the namespace set the level and mode (enforcerejects,warnwarnskubectl,auditwrites to the audit log). - The
securityContextsits on the Pod (user, group,fsGroup, seccomp) and on the container (capabilities, privilege escalation, read-only). - A dedicated ServiceAccount without a token; RBAC governs what an account may do on the API.
kubectl apply ─► API server ─► admission "PodSecurity" (labels on namespace notes)
enforce=restricted → Pod is rejected
warn=restricted → warning to kubectl
Pod securityContext: runAsNonRoot · runAsUser/Group · fsGroup · seccompProfile
└─ Container securityContext: allowPrivilegeEscalation · readOnlyRootFilesystem
capabilities.drop [ALL]
Drill 1: What a Pod May Do Today
source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag08 && cd ~/k8s-lab/tag08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag07/notes/." .
kubectl exec deploy/web -- id
kubectl exec deploy/web -- sh -c 'touch /app/x; touch /tmp/x && echo "/tmp beschreibbar"
grep -E "^(CapBnd|NoNewPrivs)" /proc/1/status'
kubectl exec deploy/web -- ls /var/run/secrets/kubernetes.io/serviceaccountExpected (abridged):
uid=1000(node) gid=1000(node) groups=1000(node)
touch: /app/x: Permission denied
/tmp beschreibbar
CapBnd: 00000000a80425fb
NoNewPrivs: 0
ca.crt
namespace
token
web runs as node because the Dockerfile wants it that
way – Kubernetes required nothing. You know the capabilities in the
bounding set from “Docker Compose hands-on”. What’s new is the token:
Kubernetes mounts a token of the ServiceAccount default
into every Pod. With it, every process in the container can authenticate
to the API:
kubectl exec deploy/web -- sh -c '
T=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
wget -qO- --no-check-certificate --header "Authorization: Bearer $T" \
https://kubernetes.default.svc/api/v1/namespaces/notes/secrets'
kubectl auth can-i list secrets --as=system:serviceaccount:notes:defaultExpected:
wget: server returned error: HTTP/1.1 403 Forbidden
command terminated with exit code 1
no
403 instead of 401: the API accepted the token and knows who is
asking – only the permission is missing (for now). If
default gets a role later, every attacker in web gets it
too. web doesn’t need the API, so the token should go.
Save your progress? Sign up for free
Drill 2: Pod Security Admission
Before you switch anything on, you ask the API server what would happen:
kubectl label --dry-run=server --overwrite ns notes \
pod-security.kubernetes.io/enforce=baseline
kubectl label --dry-run=server --overwrite ns notes \
pod-security.kubernetes.io/enforce=restrictedExpected (abridged, names, IPs and times will differ on your machine):
namespace/notes labeled (server dry run)
namespace/notes labeled (server dry run)
Warning: existing pods in namespace "notes" violate the new PodSecurity enforce level
"restricted:latest"
Warning: api-f9c7bb64f-75dcz (and 7 other pods): allowPrivilegeEscalation != false,
unrestricted capabilities, runAsNonRoot != true, seccompProfile
notes already satisfies baseline (no
privileged, no hostPath, no host namespaces).
restricted requires four more things – exactly the list in
the warning. Add the labels to k8s/namespace.yaml:
# k8s/namespace.yaml (excerpt) – state at end of Day 8
apiVersion: v1
kind: Namespace
metadata:
name: notes
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/warn: restrictedkubectl apply -f k8s/namespace.yaml
kubectl get pods -l app.kubernetes.io/name=web
kubectl rollout restart deploy/web
kubectl get rs -l app.kubernetes.io/name=web
kubectl get events --field-selector reason=FailedCreate \
-o custom-columns=MSG:.message | tail -1Expected (abridged):
namespace/notes configured
Warning: existing pods in namespace "notes" violate the new PodSecurity enforce level
"restricted:latest"
Warning: api-f9c7bb64f-75dcz (and 7 other pods): allowPrivilegeEscalation != false, …
limitrange/defaults unchanged
resourcequota/notes unchanged
NAME READY STATUS RESTARTS AGE
web-857bcf49dc-cc95h 1/1 Running 0 78s
web-857bcf49dc-xpkxw 1/1 Running 0 78s
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation !=
false (container "web" must set securityContext.allowPrivilegeEscalation=false), …
deployment.apps/web restarted
NAME DESIRED CURRENT READY AGE
web-55b6dc6795 1 0 0 5s
web-857bcf49dc 2 2 2 84s
Error creating: pods "web-55b6dc6795-xm65l" is forbidden: violates PodSecurity
"restricted:latest": allowPrivilegeEscalation != false (container "web" must set …
- Running Pods stay untouched – the check only happens on creation.
warnalready reports when the Deployment is changed;enforceonly rejects the Pod. That is why you don’t see the error inkubectl, but on the ReplicaSet (CURRENT 0) and in the events.:latestmeans: the rules of the current Kubernetes version. Withpod-security.kubernetes.io/enforce-version: v1.37you freeze them.
The rollout of web now hangs; the two old Pods keep serving. You fix that in Drill 4.
Save your progress? Sign up for free
Drill 3: api with a securityContext
api is the easiest target: a scratch image, numeric user 65532,
writes nothing. In k8s/api.yaml, add under
template.spec and on the container:
spec:
serviceAccountName: notes
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
terminationGracePeriodSeconds: 30
containers:
- name: api
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
image: notes-api:0.2.0
# … rest as on Day 7- Pod level:
runAsNonRootmakes the kubelet refuse to start if the image did run as root after all;RuntimeDefaultactivates the runtime’s seccomp profile. - Container level: no privilege gain via setuid, a read-only root file system, all capabilities dropped.
kubectl apply --dry-run=server -f k8s/api.yaml
kubectl apply -f k8s/api.yaml
kubectl get rs -l app.kubernetes.io/name=api
kubectl get events --field-selector reason=FailedCreate \
-o custom-columns=MSG:.message | grep api | tail -1Expected (abridged):
deployment.apps/api configured (server dry run)
service/api unchanged (server dry run)
deployment.apps/api configured
service/api unchanged
NAME DESIRED CURRENT READY AGE
api-6474977fbf 1 0 0 2m7s
api-f9c7bb64f 3 3 3 3m41s
Error creating: pods "api-6474977fbf-" is forbidden: error looking up service account
notes/notes: serviceaccount "notes" not found
The dry run was green, yet no Pod is created: the ServiceAccount
notes doesn’t exist yet. It belongs in
k8s/namespace.yaml – at the end of the file:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: notes
namespace: notes
automountServiceAccountToken: falsekubectl apply -f k8s/namespace.yaml
kubectl rollout status deploy/api --timeout 120s
kubectl get pod -l app.kubernetes.io/name=api \
-o jsonpath='{.items[0].spec.containers[0].volumeMounts[*].mountPath}{"\n"}'Expected:
namespace/notes unchanged
limitrange/defaults unchanged
resourcequota/notes unchanged
serviceaccount/notes created
deployment "api" successfully rolled out
/etc/notes
The ReplicaSet controller retries at growing intervals; after about
half a minute the rollout continues by itself. Only
/etc/notes is mounted in the Pod now – the token directory
is missing because automountServiceAccountToken: false is
set on the ServiceAccount. A Pod could request it again with the same
field in spec.
Save your progress? Sign up for free
Drill 4: web and the Username Trap
Add the same blocks to k8s/web.yaml, plus a
/tmp as an emptyDir – otherwise nothing at all
would be writable:
spec:
serviceAccountName: notes
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: web
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
# … image, env, ports, resources as on Day 6
volumeMounts:
- name: tmp
mountPath: /tmp
# … probes
volumes:
- name: tmp
emptyDir: {}kubectl apply -f k8s/web.yaml
sleep 15
kubectl get pods -l app.kubernetes.io/name=web
kubectl get events --field-selector reason=Failed -o custom-columns=MSG:.message \
| tail -1Expected (abridged):
deployment.apps/web configured
service/web unchanged
NAME READY STATUS RESTARTS AGE
web-79cdcdcd9c-qmw5p 0/1 CreateContainerConfigError 0 15s
web-857bcf49dc-cc95h 1/1 Running 0 4m57s
web-857bcf49dc-xpkxw 1/1 Running 0 4m57s
Error: container has runAsNonRoot and image has non-numeric user (node), cannot
verify user is non-root (pod: "web-79cdcdcd9c-qmw5p_notes(…)", container: web)
The Node image sets USER node – a name. The kubelet
doesn’t look into the image’s /etc/passwd and can’t prove
that node is not UID 0. So you state the UID yourself (it
was shown by id in Drill 1):
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
seccompProfile:
type: RuntimeDefaultkubectl apply -f k8s/web.yaml
kubectl rollout status deploy/web --timeout 120s
kubectl exec deploy/web -- sh -c 'id; touch /app/x; touch /x
touch /tmp/x && echo "/tmp beschreibbar"
grep -E "^(CapBnd|NoNewPrivs|Seccomp):" /proc/1/status
ls /var/run/secrets/kubernetes.io/serviceaccount'Expected:
deployment.apps/web configured
service/web unchanged
deployment "web" successfully rolled out
uid=1000(node) gid=1000(node) groups=1000(node)
touch: /app/x: Read-only file system
touch: /x: Read-only file system
/tmp beschreibbar
CapBnd: 0000000000000000
NoNewPrivs: 1
Seccomp: 2
ls: /var/run/secrets/kubernetes.io/serviceaccount: No such file or directory
command terminated with exit code 1
No capabilities, no privilege gain, seccomp active (2 =
filter), no token. The same error hits every image with a name in
USER – including curlimages/curl
(curl_user, UID 100) on Day 10.
Save your progress? Sign up for free
Drill 5: worker and the Missing
/tmp
worker gets the blocks from Drill 3 (user 65532 is numeric) and also
an emptyDir for /tmp. Does it really need
that? Without /tmp it runs at first – but every library
that creates a temp file fails (tried in the test):
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp',
'/var/tmp', '/usr/tmp', '/app']
With the volume (complete file:
code/tag08/notes/k8s/worker.yaml):
volumeMounts:
- name: api-key
mountPath: /etc/notes
readOnly: true
- name: tmp
mountPath: /tmp
volumes:
- name: api-key
secret:
secretName: notes-api
- name: tmp
emptyDir: {}kubectl apply -f k8s/worker.yaml
kubectl rollout status deploy/worker --timeout 90s
kubectl exec deploy/worker -- python -c \
'import tempfile, os; print(tempfile.gettempdir(), os.getuid())'Expected:
deployment.apps/worker configured
deployment "worker" successfully rolled out
/tmp 65532
An emptyDir lives as long as the Pod. With
medium: Memory it would live in RAM (like tmpfs in Docker)
and count toward the memory limit.
Save your progress? Sign up for free
Drill 6: Postgres as UID 70
Now db. First only the blocks from Drill 3 in
k8s/db.yaml:
kubectl apply -f k8s/db.yaml
sleep 20
kubectl get pod db-0
kubectl get events --field-selector involvedObject.name=db-0,reason=Failed \
-o custom-columns=MSG:.message | tail -1Expected (abridged):
service/db unchanged
statefulset.apps/db configured
NAME READY STATUS RESTARTS AGE
db-0 0/1 CreateContainerConfigError 0 19s
Error: container has runAsNonRoot and image will run as root (pod: "db-0_notes(…)",
container: postgres)
The Postgres image starts as root and only switches to
postgres in the entrypoint. The Alpine image uses UID/GID
70 for that – so start that way right away:
securityContext:
runAsNonRoot: true
runAsUser: 70
runAsGroup: 70
fsGroup: 70
seccompProfile:
type: RuntimeDefaultkubectl apply -f k8s/db.yaml
kubectl delete pod db-0
sleep 25
kubectl get pod db-0
kubectl logs db-0 --tail 2Expected (abridged):
service/db unchanged
statefulset.apps/db configured
pod "db-0" deleted from notes namespace
NAME READY STATUS RESTARTS AGE
db-0 0/1 CrashLoopBackOff 2 (5s ago) 24s
… FATAL: could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock":
Read-only file system
… LOG: database system is shut down
Warning: If a StatefulSet Pod is stuck in an error state, the controller doesn’t replace it by itself after an
apply(in the test: db-0 stayed on the old revision). The docs call this “forced rollback”: delete the broken Pod by hand.
The socket and the lock file need a writable directory. Add two
emptyDir volumes (complete file:
code/tag08/notes/k8s/db.yaml):
volumeMounts:
- name: data
mountPath: /var/lib/postgresql
- name: run
mountPath: /var/run/postgresql
- name: tmp
mountPath: /tmp
volumes:
- name: run
emptyDir: {}
- name: tmp
emptyDir: {}kubectl apply -f k8s/db.yaml
kubectl delete pod db-0
kubectl rollout status sts/db --timeout 120s
kubectl exec db-0 -- sh -c 'id; ls -ldn /var/run/postgresql /var/lib/postgresql \
/var/lib/postgresql/18/docker'
kubectl exec db-0 -- psql -U notes -tAc 'select count(*) from notes'Expected (abridged):
service/db unchanged
statefulset.apps/db configured
pod "db-0" deleted from notes namespace
partitioned roll out complete: 1 new pods have been updated...
uid=70(postgres) gid=70(postgres) groups=70(postgres)
drwxrwxrwx 3 0 0 4096 … /var/lib/postgresql
drwx------ 19 70 0 4096 … /var/lib/postgresql/18/docker
drwxrwsrwx 2 0 70 4096 … /var/run/postgresql
3
fsGroup: 70gives volumes the group 70 (plus the setgid bit): visible on theemptyDir. On the data volume it has no effect in kind – the local-path provisioner createshostPathvolumes (0777). On real block storage,fsGroupensures that UID 70 may write to a fresh volume.- The data from Day 7 is there: back then, the entrypoint script had already handed it over to UID 70.
During the drills the database was gone; api reported itself not ready. Fine in the lab.
Save your progress? Sign up for free
Drill 7: migrate – Jobs Are Immutable
migrate gets the blocks from Drill 3, nothing else (complete file:
code/tag08/notes/k8s/migrate.yaml). If the Job from Day 7
is still in the cluster (ttlSecondsAfterFinished: 600
deletes it ten minutes after it ends), apply fails. In the
test it was still there:
kubectl get job migrate
kubectl apply -f k8s/migrate.yamlExpected (abridged):
NAME STATUS COMPLETIONS DURATION AGE
migrate Complete 1/1 7s 8m28s
The Job "migrate" is invalid: spec.template: Invalid value: {…}: field is immutable
The Pod template of a Job can’t be changed after creation. On your
machine you will probably get job.batch/migrate created
instead of the error message – then the old Job has already expired. The
safe way is always: delete and create again.
kubectl delete job migrate --ignore-not-found
kubectl apply -f k8s/migrate.yaml
kubectl wait --for=condition=complete job/migrate --timeout 60s
kubectl logs job/migrateExpected (abridged):
job.batch "migrate" deleted from notes namespace
job.batch/migrate created
job.batch/migrate condition met
time=… level=INFO msg="migration complete" table=notes
Try the lock once more with a change to the fresh Job:
sed 's/notes-api:0.2.0/notes-api:0.1.0/' k8s/migrate.yaml | kubectl apply -f -Expected (abridged):
The Job "migrate" is invalid: spec.template: Invalid value: {…}: field is immutable
On Day 10, a Helm hook with before-hook-creation takes
care of this delete-and-recreate.
Save your progress? Sign up for free
Drill 8: RBAC – Role and RoleBinding
notes needs no permissions on the API. You practice how to grant them
on a test account reader that may read Pods and their logs
– nothing more:
cat > rbac-test.yaml <<'EOF'
# rbac-test.yaml – only for Day 8, Drill 8 (delete afterwards)
apiVersion: v1
kind: ServiceAccount
metadata:
name: reader
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
rules:
- apiGroups: [""]
resources: [pods, pods/log]
verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: reader-pod-reader
subjects:
- kind: ServiceAccount
name: reader
namespace: notes
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-reader
EOF
kubectl apply --dry-run=server -f rbac-test.yaml && kubectl apply -f rbac-test.yaml
SA=system:serviceaccount:notes:reader
for v in "list pods" "get pods/log" "delete pods" "get secrets" \
"list pods -n default"; do
printf '%-22s %s\n' "$v" "$(kubectl auth can-i $v --as=$SA)"
done
kubectl get pods -l app.kubernetes.io/name=db --as=$SA
kubectl get secrets --as=$SAExpected (abridged):
serviceaccount/reader created (server dry run)
role.rbac.authorization.k8s.io/pod-reader created (server dry run)
rolebinding.rbac.authorization.k8s.io/reader-pod-reader created (server dry run)
serviceaccount/reader created
role.rbac.authorization.k8s.io/pod-reader created
rolebinding.rbac.authorization.k8s.io/reader-pod-reader created
list pods yes
get pods/log yes
delete pods no
get secrets no
list pods -n default no
NAME READY STATUS RESTARTS AGE
db-0 1/1 Running 0 47s
Error from server (Forbidden): secrets is forbidden: User
"system:serviceaccount:notes:reader" cannot list resource "secrets" in API group ""
in the namespace "notes"
- A Role is a list of allowed verbs on resources in a namespace. RBAC only knows permissions, no denials – whatever is not allowed is forbidden.
- The RoleBinding links the Role to subjects (ServiceAccount, user, group). For the whole cluster, the counterparts are called ClusterRole and ClusterRoleBinding.
--aslets you ask as a different account without knowing its token. You are allowed to do that because kind’s admin context may do everything.
kubectl auth can-i --list --as=$SA shows all permissions
of the account at once. Get rid of the test account:
kubectl delete -f rbac-test.yaml && rm rbac-test.yamlExpected:
serviceaccount "reader" deleted from notes namespace
role.rbac.authorization.k8s.io "pod-reader" deleted from notes namespace
rolebinding.rbac.authorization.k8s.io "reader-pod-reader" deleted from notes namespace
Save your progress? Sign up for free
Checkpoint
kubectl get ns notes --show-labelsExpected: enforce and warn are set to
restricted.
NAME STATUS AGE LABELS
notes Active 5m44s kubernetes.io/metadata.name=notes,pod-security.kubernetes.io
/enforce=restricted,pod-security.kubernetes.io/warn=restricted
kubectl get pods -o custom-columns=NAME:.metadata.name,SA:.spec.serviceAccountNameExpected: all Pods with the ServiceAccount notes.
NAME SA
api-6474977fbf-hb4m2 notes
api-6474977fbf-sclm9 notes
api-6474977fbf-x7fjk notes
db-0 notes
migrate-b89df notes
web-df4dcfbcf-7tlbs notes
web-df4dcfbcf-h9q44 notes
worker-b4b948756-5wqmf notes
kubectl exec deploy/web -- id
kubectl exec deploy/web -- touch /xExpected: web runs as UID 1000, and the file system is read-only.
uid=1000(node) gid=1000(node) groups=1000(node)
touch: /x: Read-only file system
command terminated with exit code 1
kubectl exec deploy/web -- ls /var/run/secrets/kubernetes.ioExpected: no ServiceAccount token in the Pod.
ls: /var/run/secrets/kubernetes.io: No such file or directory
command terminated with exit code 1
kubectl exec db-0 -- idExpected: Postgres runs as UID 70.
uid=70(postgres) gid=70(postgres) groups=70(postgres)
Challenge
In the namespace notes, start a Pod debug
with busybox:1.37 that runs id,
touch /x and wget -qO- http://api:8080/version
once – and is admitted by restricted.
Hint: busybox runs as root since the image has no
USER; kubectl run without security settings is
rejected.
Solution
kubectl run debug --image=busybox:1.37 --restart=Never -- id
cat > debug.yaml <<'EOF'
# debug.yaml – Challenge Day 8: busybox under restricted
apiVersion: v1
kind: Pod
metadata:
name: debug
spec:
serviceAccountName: notes
restartPolicy: Never
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: debug
image: busybox:1.37
command: [sh, -c, "id; touch /x; wget -qO- http://api:8080/version"]
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
EOF
kubectl apply -f debug.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/debug --timeout 60s
kubectl logs debug
kubectl delete -f debug.yamlExpected (abridged):
Error from server (Forbidden): pods "debug" is forbidden: violates PodSecurity
"restricted:latest": allowPrivilegeEscalation != false (container "debug" must set …
pod/debug created
pod/debug condition met
uid=65532 gid=65532 groups=65532
touch: /x: Read-only file system
0.2.0
pod "debug" deleted from notes namespace
The UID can be anything as long as it isn’t 0 – busybox needs no
entry in /etc/passwd. The LimitRange from Day 6 adds
requests and limits. A more convenient way to debug running Pods follows
on Day 11 with kubectl debug.
Save your progress? Sign up for free
Enterprise Note
- Pod Security Admission is deliberately coarse: three fixed levels per namespace. Finer rules (allowed registries, mandatory labels, signatures) are enforced by a policy engine – see “Kyverno & NetworkPolicies hands-on”.
- New namespaces get at least
warn/auditonrestrictedvia a template (or policy), so that violations stand out before anyone turns onenforce. - RBAC according to the principle of least privilege: no
ClusterRoleBindings to
cluster-adminfor humans in day-to-day work, groups from the identity provider instead of individuals, checked regularly withkubectl auth can-i --list. - Build images with a numeric
USER(“Dockerfiles hands-on”) – that spares you maintainingrunAsUserin every manifest. - Runtime detection (e.g. Falco) and a custom seccomp profile are the
next steps;
RuntimeDefaultis the minimum.