Skip to content
Hands-on IT IT Drills

Day 8: Security in the Pod

Goal: All notes Pods run under Pod Security “restricted”: without root, without capabilities, with a read-only file system and without a token for the Kubernetes API. · Duration: approx. 90 min · Prerequisite: Day 7

Briefing

  • So far, whatever the image and runtime specify applies: capabilities, a writable file system and the token of the ServiceAccount default in every container.
  • Pod Security Admission checks every new Pod against one of three levels: privileged, baseline, restricted. Labels on the namespace set the level and mode (enforce rejects, warn warns kubectl, audit writes to the audit log).
  • The securityContext sits on the Pod (user, group, fsGroup, seccomp) and on the container (capabilities, privilege escalation, read-only).
  • A dedicated ServiceAccount without a token; RBAC governs what an account may do on the API.
 kubectl apply ─► API server ─► admission "PodSecurity" (labels on namespace notes)
                                  enforce=restricted → Pod is rejected
                                  warn=restricted    → warning to kubectl
 Pod  securityContext: runAsNonRoot · runAsUser/Group · fsGroup · seccompProfile
  └─ Container securityContext: allowPrivilegeEscalation · readOnlyRootFilesystem
                                capabilities.drop [ALL]

Drill 1: What a Pod May Do Today

source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag08 && cd ~/k8s-lab/tag08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag07/notes/." .
kubectl exec deploy/web -- id
kubectl exec deploy/web -- sh -c 'touch /app/x; touch /tmp/x && echo "/tmp beschreibbar"
  grep -E "^(CapBnd|NoNewPrivs)" /proc/1/status'
kubectl exec deploy/web -- ls /var/run/secrets/kubernetes.io/serviceaccount

Expected (abridged):

uid=1000(node) gid=1000(node) groups=1000(node)
touch: /app/x: Permission denied
/tmp beschreibbar
CapBnd: 00000000a80425fb
NoNewPrivs: 0
ca.crt
namespace
token

web runs as node because the Dockerfile wants it that way – Kubernetes required nothing. You know the capabilities in the bounding set from “Docker Compose hands-on”. What’s new is the token: Kubernetes mounts a token of the ServiceAccount default into every Pod. With it, every process in the container can authenticate to the API:

kubectl exec deploy/web -- sh -c '
  T=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
  wget -qO- --no-check-certificate --header "Authorization: Bearer $T" \
    https://kubernetes.default.svc/api/v1/namespaces/notes/secrets'
kubectl auth can-i list secrets --as=system:serviceaccount:notes:default

Expected:

wget: server returned error: HTTP/1.1 403 Forbidden
command terminated with exit code 1
no

403 instead of 401: the API accepted the token and knows who is asking – only the permission is missing (for now). If default gets a role later, every attacker in web gets it too. web doesn’t need the API, so the token should go.

Save your progress? Sign up for free

Drill 2: Pod Security Admission

Before you switch anything on, you ask the API server what would happen:

kubectl label --dry-run=server --overwrite ns notes \
  pod-security.kubernetes.io/enforce=baseline
kubectl label --dry-run=server --overwrite ns notes \
  pod-security.kubernetes.io/enforce=restricted

Expected (abridged, names, IPs and times will differ on your machine):

namespace/notes labeled (server dry run)
namespace/notes labeled (server dry run)
Warning: existing pods in namespace "notes" violate the new PodSecurity enforce level
"restricted:latest"
Warning: api-f9c7bb64f-75dcz (and 7 other pods): allowPrivilegeEscalation != false,
unrestricted capabilities, runAsNonRoot != true, seccompProfile

notes already satisfies baseline (no privileged, no hostPath, no host namespaces). restricted requires four more things – exactly the list in the warning. Add the labels to k8s/namespace.yaml:

# k8s/namespace.yaml (excerpt) – state at end of Day 8
apiVersion: v1
kind: Namespace
metadata:
  name: notes
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted
kubectl apply -f k8s/namespace.yaml
kubectl get pods -l app.kubernetes.io/name=web
kubectl rollout restart deploy/web
kubectl get rs -l app.kubernetes.io/name=web
kubectl get events --field-selector reason=FailedCreate \
  -o custom-columns=MSG:.message | tail -1

Expected (abridged):

namespace/notes configured
Warning: existing pods in namespace "notes" violate the new PodSecurity enforce level
"restricted:latest"
Warning: api-f9c7bb64f-75dcz (and 7 other pods): allowPrivilegeEscalation != false, …
limitrange/defaults unchanged
resourcequota/notes unchanged
NAME                   READY   STATUS    RESTARTS   AGE
web-857bcf49dc-cc95h   1/1     Running   0          78s
web-857bcf49dc-xpkxw   1/1     Running   0          78s
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation !=
false (container "web" must set securityContext.allowPrivilegeEscalation=false), …
deployment.apps/web restarted
NAME             DESIRED   CURRENT   READY   AGE
web-55b6dc6795   1         0         0       5s
web-857bcf49dc   2         2         2       84s
Error creating: pods "web-55b6dc6795-xm65l" is forbidden: violates PodSecurity
"restricted:latest": allowPrivilegeEscalation != false (container "web" must set …
  • Running Pods stay untouched – the check only happens on creation.
  • warn already reports when the Deployment is changed; enforce only rejects the Pod. That is why you don’t see the error in kubectl, but on the ReplicaSet (CURRENT 0) and in the events.
  • :latest means: the rules of the current Kubernetes version. With pod-security.kubernetes.io/enforce-version: v1.37 you freeze them.

The rollout of web now hangs; the two old Pods keep serving. You fix that in Drill 4.

Save your progress? Sign up for free

Drill 3: api with a securityContext

api is the easiest target: a scratch image, numeric user 65532, writes nothing. In k8s/api.yaml, add under template.spec and on the container:

    spec:
      serviceAccountName: notes
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ALL]
          image: notes-api:0.2.0
          # … rest as on Day 7
  • Pod level: runAsNonRoot makes the kubelet refuse to start if the image did run as root after all; RuntimeDefault activates the runtime’s seccomp profile.
  • Container level: no privilege gain via setuid, a read-only root file system, all capabilities dropped.
kubectl apply --dry-run=server -f k8s/api.yaml
kubectl apply -f k8s/api.yaml
kubectl get rs -l app.kubernetes.io/name=api
kubectl get events --field-selector reason=FailedCreate \
  -o custom-columns=MSG:.message | grep api | tail -1

Expected (abridged):

deployment.apps/api configured (server dry run)
service/api unchanged (server dry run)
deployment.apps/api configured
service/api unchanged
NAME             DESIRED   CURRENT   READY   AGE
api-6474977fbf   1         0         0       2m7s
api-f9c7bb64f    3         3         3       3m41s
Error creating: pods "api-6474977fbf-" is forbidden: error looking up service account
notes/notes: serviceaccount "notes" not found

The dry run was green, yet no Pod is created: the ServiceAccount notes doesn’t exist yet. It belongs in k8s/namespace.yaml – at the end of the file:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: notes
  namespace: notes
automountServiceAccountToken: false
kubectl apply -f k8s/namespace.yaml
kubectl rollout status deploy/api --timeout 120s
kubectl get pod -l app.kubernetes.io/name=api \
  -o jsonpath='{.items[0].spec.containers[0].volumeMounts[*].mountPath}{"\n"}'

Expected:

namespace/notes unchanged
limitrange/defaults unchanged
resourcequota/notes unchanged
serviceaccount/notes created
deployment "api" successfully rolled out
/etc/notes

The ReplicaSet controller retries at growing intervals; after about half a minute the rollout continues by itself. Only /etc/notes is mounted in the Pod now – the token directory is missing because automountServiceAccountToken: false is set on the ServiceAccount. A Pod could request it again with the same field in spec.

Save your progress? Sign up for free

Drill 4: web and the Username Trap

Add the same blocks to k8s/web.yaml, plus a /tmp as an emptyDir – otherwise nothing at all would be writable:

    spec:
      serviceAccountName: notes
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: web
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ALL]
          # … image, env, ports, resources as on Day 6
          volumeMounts:
            - name: tmp
              mountPath: /tmp
          # … probes
      volumes:
        - name: tmp
          emptyDir: {}
kubectl apply -f k8s/web.yaml
sleep 15
kubectl get pods -l app.kubernetes.io/name=web
kubectl get events --field-selector reason=Failed -o custom-columns=MSG:.message \
  | tail -1

Expected (abridged):

deployment.apps/web configured
service/web unchanged
NAME                   READY   STATUS                       RESTARTS   AGE
web-79cdcdcd9c-qmw5p   0/1     CreateContainerConfigError   0          15s
web-857bcf49dc-cc95h   1/1     Running                      0          4m57s
web-857bcf49dc-xpkxw   1/1     Running                      0          4m57s
Error: container has runAsNonRoot and image has non-numeric user (node), cannot
verify user is non-root (pod: "web-79cdcdcd9c-qmw5p_notes(…)", container: web)

The Node image sets USER node – a name. The kubelet doesn’t look into the image’s /etc/passwd and can’t prove that node is not UID 0. So you state the UID yourself (it was shown by id in Drill 1):

      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
kubectl apply -f k8s/web.yaml
kubectl rollout status deploy/web --timeout 120s
kubectl exec deploy/web -- sh -c 'id; touch /app/x; touch /x
  touch /tmp/x && echo "/tmp beschreibbar"
  grep -E "^(CapBnd|NoNewPrivs|Seccomp):" /proc/1/status
  ls /var/run/secrets/kubernetes.io/serviceaccount'

Expected:

deployment.apps/web configured
service/web unchanged
deployment "web" successfully rolled out
uid=1000(node) gid=1000(node) groups=1000(node)
touch: /app/x: Read-only file system
touch: /x: Read-only file system
/tmp beschreibbar
CapBnd: 0000000000000000
NoNewPrivs: 1
Seccomp:    2
ls: /var/run/secrets/kubernetes.io/serviceaccount: No such file or directory
command terminated with exit code 1

No capabilities, no privilege gain, seccomp active (2 = filter), no token. The same error hits every image with a name in USER – including curlimages/curl (curl_user, UID 100) on Day 10.

Save your progress? Sign up for free

Drill 5: worker and the Missing /tmp

worker gets the blocks from Drill 3 (user 65532 is numeric) and also an emptyDir for /tmp. Does it really need that? Without /tmp it runs at first – but every library that creates a temp file fails (tried in the test):

FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp',
'/var/tmp', '/usr/tmp', '/app']

With the volume (complete file: code/tag08/notes/k8s/worker.yaml):

          volumeMounts:
            - name: api-key
              mountPath: /etc/notes
              readOnly: true
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: api-key
          secret:
            secretName: notes-api
        - name: tmp
          emptyDir: {}
kubectl apply -f k8s/worker.yaml
kubectl rollout status deploy/worker --timeout 90s
kubectl exec deploy/worker -- python -c \
  'import tempfile, os; print(tempfile.gettempdir(), os.getuid())'

Expected:

deployment.apps/worker configured
deployment "worker" successfully rolled out
/tmp 65532

An emptyDir lives as long as the Pod. With medium: Memory it would live in RAM (like tmpfs in Docker) and count toward the memory limit.

Save your progress? Sign up for free

Drill 6: Postgres as UID 70

Now db. First only the blocks from Drill 3 in k8s/db.yaml:

kubectl apply -f k8s/db.yaml
sleep 20
kubectl get pod db-0
kubectl get events --field-selector involvedObject.name=db-0,reason=Failed \
  -o custom-columns=MSG:.message | tail -1

Expected (abridged):

service/db unchanged
statefulset.apps/db configured
NAME   READY   STATUS                       RESTARTS   AGE
db-0   0/1     CreateContainerConfigError   0          19s
Error: container has runAsNonRoot and image will run as root (pod: "db-0_notes(…)",
container: postgres)

The Postgres image starts as root and only switches to postgres in the entrypoint. The Alpine image uses UID/GID 70 for that – so start that way right away:

      securityContext:
        runAsNonRoot: true
        runAsUser: 70
        runAsGroup: 70
        fsGroup: 70
        seccompProfile:
          type: RuntimeDefault
kubectl apply -f k8s/db.yaml
kubectl delete pod db-0
sleep 25
kubectl get pod db-0
kubectl logs db-0 --tail 2

Expected (abridged):

service/db unchanged
statefulset.apps/db configured
pod "db-0" deleted from notes namespace
NAME   READY   STATUS             RESTARTS     AGE
db-0   0/1     CrashLoopBackOff   2 (5s ago)   24s
… FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock":
Read-only file system
… LOG:  database system is shut down

Warning: If a StatefulSet Pod is stuck in an error state, the controller doesn’t replace it by itself after an apply (in the test: db-0 stayed on the old revision). The docs call this “forced rollback”: delete the broken Pod by hand.

The socket and the lock file need a writable directory. Add two emptyDir volumes (complete file: code/tag08/notes/k8s/db.yaml):

          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql
            - name: run
              mountPath: /var/run/postgresql
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: run
          emptyDir: {}
        - name: tmp
          emptyDir: {}
kubectl apply -f k8s/db.yaml
kubectl delete pod db-0
kubectl rollout status sts/db --timeout 120s
kubectl exec db-0 -- sh -c 'id; ls -ldn /var/run/postgresql /var/lib/postgresql \
  /var/lib/postgresql/18/docker'
kubectl exec db-0 -- psql -U notes -tAc 'select count(*) from notes'

Expected (abridged):

service/db unchanged
statefulset.apps/db configured
pod "db-0" deleted from notes namespace
partitioned roll out complete: 1 new pods have been updated...
uid=70(postgres) gid=70(postgres) groups=70(postgres)
drwxrwxrwx    3 0        0             4096 … /var/lib/postgresql
drwx------   19 70       0             4096 … /var/lib/postgresql/18/docker
drwxrwsrwx    2 0        70            4096 … /var/run/postgresql
3
  • fsGroup: 70 gives volumes the group 70 (plus the setgid bit): visible on the emptyDir. On the data volume it has no effect in kind – the local-path provisioner creates hostPath volumes (0777). On real block storage, fsGroup ensures that UID 70 may write to a fresh volume.
  • The data from Day 7 is there: back then, the entrypoint script had already handed it over to UID 70.

During the drills the database was gone; api reported itself not ready. Fine in the lab.

Save your progress? Sign up for free

Drill 7: migrate – Jobs Are Immutable

migrate gets the blocks from Drill 3, nothing else (complete file: code/tag08/notes/k8s/migrate.yaml). If the Job from Day 7 is still in the cluster (ttlSecondsAfterFinished: 600 deletes it ten minutes after it ends), apply fails. In the test it was still there:

kubectl get job migrate
kubectl apply -f k8s/migrate.yaml

Expected (abridged):

NAME      STATUS     COMPLETIONS   DURATION   AGE
migrate   Complete   1/1           7s         8m28s
The Job "migrate" is invalid: spec.template: Invalid value: {…}: field is immutable

The Pod template of a Job can’t be changed after creation. On your machine you will probably get job.batch/migrate created instead of the error message – then the old Job has already expired. The safe way is always: delete and create again.

kubectl delete job migrate --ignore-not-found
kubectl apply -f k8s/migrate.yaml
kubectl wait --for=condition=complete job/migrate --timeout 60s
kubectl logs job/migrate

Expected (abridged):

job.batch "migrate" deleted from notes namespace
job.batch/migrate created
job.batch/migrate condition met
time=… level=INFO msg="migration complete" table=notes

Try the lock once more with a change to the fresh Job:

sed 's/notes-api:0.2.0/notes-api:0.1.0/' k8s/migrate.yaml | kubectl apply -f -

Expected (abridged):

The Job "migrate" is invalid: spec.template: Invalid value: {…}: field is immutable

On Day 10, a Helm hook with before-hook-creation takes care of this delete-and-recreate.

Save your progress? Sign up for free

Drill 8: RBAC – Role and RoleBinding

notes needs no permissions on the API. You practice how to grant them on a test account reader that may read Pods and their logs – nothing more:

cat > rbac-test.yaml <<'EOF'
# rbac-test.yaml – only for Day 8, Drill 8 (delete afterwards)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: reader
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
rules:
  - apiGroups: [""]
    resources: [pods, pods/log]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: reader-pod-reader
subjects:
  - kind: ServiceAccount
    name: reader
    namespace: notes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader
EOF
kubectl apply --dry-run=server -f rbac-test.yaml && kubectl apply -f rbac-test.yaml
SA=system:serviceaccount:notes:reader
for v in "list pods" "get pods/log" "delete pods" "get secrets" \
  "list pods -n default"; do
  printf '%-22s %s\n' "$v" "$(kubectl auth can-i $v --as=$SA)"
done
kubectl get pods -l app.kubernetes.io/name=db --as=$SA
kubectl get secrets --as=$SA

Expected (abridged):

serviceaccount/reader created (server dry run)
role.rbac.authorization.k8s.io/pod-reader created (server dry run)
rolebinding.rbac.authorization.k8s.io/reader-pod-reader created (server dry run)
serviceaccount/reader created
role.rbac.authorization.k8s.io/pod-reader created
rolebinding.rbac.authorization.k8s.io/reader-pod-reader created
list pods              yes
get pods/log           yes
delete pods            no
get secrets            no
list pods -n default   no
NAME   READY   STATUS    RESTARTS   AGE
db-0   1/1     Running   0          47s
Error from server (Forbidden): secrets is forbidden: User
"system:serviceaccount:notes:reader" cannot list resource "secrets" in API group ""
in the namespace "notes"
  • A Role is a list of allowed verbs on resources in a namespace. RBAC only knows permissions, no denials – whatever is not allowed is forbidden.
  • The RoleBinding links the Role to subjects (ServiceAccount, user, group). For the whole cluster, the counterparts are called ClusterRole and ClusterRoleBinding.
  • --as lets you ask as a different account without knowing its token. You are allowed to do that because kind’s admin context may do everything.

kubectl auth can-i --list --as=$SA shows all permissions of the account at once. Get rid of the test account:

kubectl delete -f rbac-test.yaml && rm rbac-test.yaml

Expected:

serviceaccount "reader" deleted from notes namespace
role.rbac.authorization.k8s.io "pod-reader" deleted from notes namespace
rolebinding.rbac.authorization.k8s.io "reader-pod-reader" deleted from notes namespace

Save your progress? Sign up for free

Checkpoint

kubectl get ns notes --show-labels

Expected: enforce and warn are set to restricted.

NAME    STATUS   AGE     LABELS
notes   Active   5m44s   kubernetes.io/metadata.name=notes,pod-security.kubernetes.io
/enforce=restricted,pod-security.kubernetes.io/warn=restricted
kubectl get pods -o custom-columns=NAME:.metadata.name,SA:.spec.serviceAccountName

Expected: all Pods with the ServiceAccount notes.

NAME                     SA
api-6474977fbf-hb4m2     notes
api-6474977fbf-sclm9     notes
api-6474977fbf-x7fjk     notes
db-0                     notes
migrate-b89df            notes
web-df4dcfbcf-7tlbs      notes
web-df4dcfbcf-h9q44      notes
worker-b4b948756-5wqmf   notes
kubectl exec deploy/web -- id
kubectl exec deploy/web -- touch /x

Expected: web runs as UID 1000, and the file system is read-only.

uid=1000(node) gid=1000(node) groups=1000(node)
touch: /x: Read-only file system
command terminated with exit code 1
kubectl exec deploy/web -- ls /var/run/secrets/kubernetes.io

Expected: no ServiceAccount token in the Pod.

ls: /var/run/secrets/kubernetes.io: No such file or directory
command terminated with exit code 1
kubectl exec db-0 -- id

Expected: Postgres runs as UID 70.

uid=70(postgres) gid=70(postgres) groups=70(postgres)

Challenge

In the namespace notes, start a Pod debug with busybox:1.37 that runs id, touch /x and wget -qO- http://api:8080/version once – and is admitted by restricted.

Hint: busybox runs as root since the image has no USER; kubectl run without security settings is rejected.

Solution

kubectl run debug --image=busybox:1.37 --restart=Never -- id
cat > debug.yaml <<'EOF'
# debug.yaml – Challenge Day 8: busybox under restricted
apiVersion: v1
kind: Pod
metadata:
  name: debug
spec:
  serviceAccountName: notes
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    runAsUser: 65532
    runAsGroup: 65532
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: debug
      image: busybox:1.37
      command: [sh, -c, "id; touch /x; wget -qO- http://api:8080/version"]
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: [ALL]
EOF
kubectl apply -f debug.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/debug --timeout 60s
kubectl logs debug
kubectl delete -f debug.yaml

Expected (abridged):

Error from server (Forbidden): pods "debug" is forbidden: violates PodSecurity
"restricted:latest": allowPrivilegeEscalation != false (container "debug" must set …
pod/debug created
pod/debug condition met
uid=65532 gid=65532 groups=65532
touch: /x: Read-only file system
0.2.0
pod "debug" deleted from notes namespace

The UID can be anything as long as it isn’t 0 – busybox needs no entry in /etc/passwd. The LimitRange from Day 6 adds requests and limits. A more convenient way to debug running Pods follows on Day 11 with kubectl debug.

Save your progress? Sign up for free

Enterprise Note

  • Pod Security Admission is deliberately coarse: three fixed levels per namespace. Finer rules (allowed registries, mandatory labels, signatures) are enforced by a policy engine – see “Kyverno & NetworkPolicies hands-on”.
  • New namespaces get at least warn/audit on restricted via a template (or policy), so that violations stand out before anyone turns on enforce.
  • RBAC according to the principle of least privilege: no ClusterRoleBindings to cluster-admin for humans in day-to-day work, groups from the identity provider instead of individuals, checked regularly with kubectl auth can-i --list.
  • Build images with a numeric USER (“Dockerfiles hands-on”) – that spares you maintaining runAsUser in every manifest.
  • Runtime detection (e.g. Falco) and a custom seccomp profile are the next steps; RuntimeDefault is the minimum.