Skip to content
Hands-on IT IT Drills

Day 11: RBAC and Pod Security

Goal: You read a Forbidden message down to the last word, check permissions with auth can-i and find Pods that Pod Security Admission rejected. · Duration: approx. 60 min · Prerequisite: Day 3

Briefing

  • An RBAC message names everything: who (User "system:serviceaccount:ns:name"), verb, resource, API group and scope (in the namespace … or at the cluster scope).
  • kubectl auth can-i … --as=… checks permissions without starting the Pod.
  • Pod Security Admission (PSA) checks Pods. A Deployment is accepted, but its Pods are rejected – the event is on the ReplicaSet or DaemonSet.
  • RKE2 sets enforce: privileged cluster-wide; restrictions come via namespace labels.
Fault Symptom Where the answer is
Binding in the wrong namespace … in the namespace get rolebinding -A
Role instead of ClusterRole at the cluster scope Cluster-wide resource?
PSA baseline DaemonSet CURRENT 0 Event FailedCreate
PSA restricted Deployment 0/1, no Pods Event on the ReplicaSet

Drill 1: The RoleBinding in the wrong namespace

Break it: A Pod that lists Pods with kubectl. The RoleBinding was copied from a template:

# on cp1
mkdir -p ~/debug/tag11 && cd ~/debug/tag11
kubectl create namespace tag11
cat > lister.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: app, namespace: tag11}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: pod-reader, namespace: tag11}
rules:
- apiGroups: [""]
  resources: [pods]
  verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: pod-reader, namespace: default}
subjects:
- {kind: ServiceAccount, name: app, namespace: tag11}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: pod-reader}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: pod-lister, namespace: tag11}
spec:
  replicas: 1
  selector: {matchLabels: {app: pod-lister}}
  template:
    metadata: {labels: {app: pod-lister}}
    spec:
      serviceAccountName: app
      containers:
      - name: kubectl
        image: registry.k8s.io/kubectl:v1.37.2
        args: [get, pods, -n, tag11, --watch]
EOF
kubectl apply -f lister.yaml

Symptom (after about 20 seconds):

# on cp1
kubectl -n tag11 get pods
kubectl -n tag11 logs deploy/pod-lister --tail=1

Expected (line wrapped):

NAME                          READY   STATUS    RESTARTS      AGE
pod-lister-754f7856b7-qt7wh   1/1     Running   2 (12s ago)   20s
Error from server (Forbidden): pods is forbidden: User
  "system:serviceaccount:tag11:app" cannot list resource "pods" in API group "" in
  the namespace "tag11"

Investigate: Read the message: who? The ServiceAccount app from tag11. What? list pods, core group "". Where? tag11. Now check the permissions for exactly that and look for the bindings that name the account:

# on cp1
kubectl auth can-i list pods -n tag11 --as=system:serviceaccount:tag11:app
kubectl auth can-i list pods -n default --as=system:serviceaccount:tag11:app
kubectl get rolebinding -A -o custom-columns=NS:.metadata.namespace,\
NAME:.metadata.name,ROLE:.roleRef.name,SA:.subjects[*].name | grep -E '^NS|app' \
  | tr -s ' '

Expected:

no
no - RBAC: role.rbac.authorization.k8s.io "pod-reader" not found
NS NAME ROLE SA
default pod-reader pod-reader app

Cause: The RoleBinding lives in default and refers there to a Role pod-reader that only exists in tag11. A RoleBinding only applies in its own namespace and binds either a Role from that namespace or a ClusterRole. The second can-i line even gives this away.

Fix:

# on cp1
kubectl -n default delete rolebinding pod-reader
kubectl -n tag11 create rolebinding pod-reader --role=pod-reader \
  --serviceaccount=tag11:app
kubectl -n tag11 rollout restart deploy/pod-lister
kubectl -n tag11 rollout status deploy/pod-lister

Verify:

# on cp1
kubectl auth can-i list pods -n tag11 --as=system:serviceaccount:tag11:app
kubectl -n tag11 logs deploy/pod-lister | head -2

Expected:

yes
NAME                          READY   STATUS        RESTARTS      AGE
pod-lister-66f9bd75c5-vhcqz   1/1     Running       0             2s

Why the restart? The container had exited with code 1 on every attempt and was in backoff (Day 5). Permissions take effect immediately, the backoff does not reset.

Save your progress? Sign up for free

Drill 2: Role instead of ClusterRole

Break it: The same ServiceAccount should read nodes:

# on cp1
cat > nodes.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: node-reader, namespace: tag11}
rules:
- apiGroups: [""]
  resources: [nodes]
  verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: node-reader, namespace: tag11}
subjects:
- {kind: ServiceAccount, name: app, namespace: tag11}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: node-reader}
EOF
kubectl apply -f nodes.yaml

Symptom:

# on cp1
kubectl get nodes --as=system:serviceaccount:tag11:app

Expected (line wrapped):

Error from server (Forbidden): nodes is forbidden: User
  "system:serviceaccount:tag11:app" cannot list resource "nodes" in API group "" at
  the cluster scope

Investigate:

# on cp1
kubectl auth can-i list nodes --as=system:serviceaccount:tag11:app
kubectl auth can-i list nodes -n tag11 --as=system:serviceaccount:tag11:app
kubectl api-resources | grep -E '^NAME|^nodes ' | tr -s ' '

Expected:

Warning: resource 'nodes' is not namespace scoped

no
Warning: resource 'nodes' is not namespace scoped

yes
NAME SHORTNAMES APIVERSION NAMESPACED KIND
nodes no v1 false Node
nodes metrics.k8s.io/v1beta1 false NodeMetrics

Cause: at the cluster scope – nodes do not belong to any namespace (NAMESPACED false). A Role can name them, but it only applies inside tag11, and there are no nodes there. The yes with -n tag11 is a trap: can-i checks the rule formally and only warns.

Fix:

# on cp1
kubectl delete -f nodes.yaml
kubectl create clusterrole tag11-node-reader --verb=get,list,watch --resource=nodes
kubectl create clusterrolebinding tag11-node-reader \
  --clusterrole=tag11-node-reader --serviceaccount=tag11:app

Verify:

# on cp1
kubectl get nodes --as=system:serviceaccount:tag11:app

Expected:

NAME      STATUS   ROLES                AGE    VERSION
cp1       Ready    control-plane,etcd   148m   v1.37.2+rke2r1
worker1   Ready    <none>               145m   v1.37.2+rke2r1
worker2   Ready    <none>               144m   v1.37.2+rke2r1

Save your progress? Sign up for free

Drill 3: All permissions at a glance

# on cp1
kubectl auth can-i --list -n tag11 --as=system:serviceaccount:tag11:app | head -6 \
  | cut -c1-88
kubectl auth whoami | cut -c1-88

Expected (abridged):

Resources                                       Non-Resource URLs                      R
selfsubjectreviews.authentication.k8s.io        []                                     [
selfsubjectaccessreviews.authorization.k8s.io   []                                     [
selfsubjectrulesreviews.authorization.k8s.io    []                                     [
nodes                                           []                                     [
pods                                            []                                     [
ATTRIBUTE                                           VALUE
Username                                            system:admin
Groups                                              [system:masters system:authenticated
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=7d22b684b588198df48b975e

Why? --list shows the sum of all bindings for the namespace – including those via ClusterRoleBindings. auth whoami answers who you are right now: the RKE2 kubeconfig is system:admin in system:masters, and that group bypasses RBAC entirely. That is why RBAC errors never show up when you test as admin.

Save your progress? Sign up for free

Drill 4: Pod Security baseline and the hostPath

Break it: A log collector as a DaemonSet in a namespace with baseline:

# on cp1
kubectl create namespace tag11-logs
kubectl label namespace tag11-logs pod-security.kubernetes.io/enforce=baseline
cat > sammler.yaml <<'EOF'
apiVersion: apps/v1
kind: DaemonSet
metadata: {name: log-sammler, namespace: tag11-logs}
spec:
  selector: {matchLabels: {app: log-sammler}}
  template:
    metadata: {labels: {app: log-sammler}}
    spec:
      containers:
      - name: sammler
        image: busybox:1.37
        command: [sh, -c, 'tail -F /host-logs/syslog']
        volumeMounts: [{name: logs, mountPath: /host-logs, readOnly: true}]
      volumes:
      - name: logs
        hostPath: {path: /var/log}
EOF
kubectl apply -f sammler.yaml

Expected:

Warning: would violate PodSecurity "baseline:latest": hostPath volumes (volume "logs")
daemonset.apps/log-sammler created

The warning already appears on apply – the DaemonSet itself is created anyway.

Symptom:

# on cp1
kubectl -n tag11-logs get ds log-sammler | cut -c1-72
kubectl -n tag11-logs get pods

Expected:

NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR
log-sammler   3         0         0       0            0           <none>
No resources found in tag11-logs namespace.

Investigate:

# on cp1
kubectl -n tag11-logs events --types=Warning | tail -1
kubectl get ns tag11-logs -L pod-security.kubernetes.io/enforce

Expected (line wrapped):

1s (x2 over 3s)   Warning   FailedCreate   DaemonSet/log-sammler   (combined from
  similar events): Error creating: pods "log-sammler-vcmsj" is forbidden: violates
  PodSecurity "baseline:latest": hostPath volumes (volume "logs")
NAME         STATUS   AGE   ENFORCE
tag11-logs   Active   7s    baseline

Cause: baseline forbids hostPath. But a log collector needs exactly that. This is not a bug in the Pod but a decision about the namespace: system-level services get their own namespace with privileged.

Fix:

# on cp1
kubectl label --overwrite namespace tag11-logs \
  pod-security.kubernetes.io/enforce=privileged

Verify:

# on cp1
kubectl -n tag11-logs rollout status ds/log-sammler
kubectl -n tag11-logs get ds log-sammler | cut -c1-72

Expected:

daemon set "log-sammler" successfully rolled out
NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR
log-sammler   3         3         3       3            3           <none>

The DaemonSet controller keeps retrying with backoff; depending on the number of failed attempts, this takes up to a minute. No restart is needed.

Talos: There, baseline is the cluster-wide default – the same log collector fails without any label (Day 19).

Save your progress? Sign up for free

Drill 5: restricted – accepted, yet rejected

Break it:

# on cp1
kubectl create namespace tag11-streng
kubectl label namespace tag11-streng pod-security.kubernetes.io/enforce=restricted
kubectl -n tag11-streng create deployment web --image=nginxinc/nginx-unprivileged:1.29

Expected (abridged):

namespace/tag11-streng created
namespace/tag11-streng labeled
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation !=
  false (container "nginx-unprivileged" must set ...
deployment.apps/web created

Symptom:

# on cp1
kubectl -n tag11-streng get deploy web
kubectl -n tag11-streng get pods

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    0/1     0            0           4s
No resources found in tag11-streng namespace.

Investigate: As on Day 3 – no Pod, so go one level up:

# on cp1
kubectl -n tag11-streng events --types=Warning | tail -1 | fold -w 86

Expected:

10s (x4 over 28s)   Warning   FailedCreate   ReplicaSet/web-86c6bddd97   (combined fro
m similar events): Error creating: pods "web-86c6bddd97-hmhpn" is forbidden: violates
PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx-u
nprivileged" must set securityContext.allowPrivilegeEscalation=false), unrestricted ca
pabilities (container "nginx-unprivileged" must set securityContext.capabilities.drop=
["ALL"]), runAsNonRoot != true (pod or container "nginx-unprivileged" must set securit
yContext.runAsNonRoot=true), seccompProfile (pod or container "nginx-unprivileged" mus
t set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Cause: restricted requires four things that hardly any manifest brings along on its own: allowPrivilegeEscalation: false, capabilities.drop: [ALL], runAsNonRoot: true and a seccompProfile. The message lists each point separately.

Fix:

# on cp1
kubectl -n tag11-streng patch deploy web -p '
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
      - name: nginx-unprivileged
        securityContext:
          allowPrivilegeEscalation: false
          capabilities: {drop: [ALL]}'
kubectl -n tag11-streng rollout status deploy/web

Verify:

# on cp1
kubectl -n tag11-streng get pods

Expected:

NAME                   READY   STATUS    RESTARTS   AGE
web-6c88d4d554-xrcqn   1/1     Running   0          2s

Why does runAsNonRoot work here? nginx-unprivileged sets the numeric user 101 in the image. With busybox (root) you would also need runAsUser.

Save your progress? Sign up for free

Drill 6: Warn first, then enforce

Before you tighten a namespace, you ask the API server what would happen – without changing anything:

# on cp1
kubectl label --dry-run=server --overwrite namespace tag11-logs \
  pod-security.kubernetes.io/enforce=baseline
kubectl label --dry-run=server --overwrite namespace tag11 \
  pod-security.kubernetes.io/enforce=restricted

Expected (line wrapped):

Warning: existing pods in namespace "tag11-logs" violate the new PodSecurity enforce
  level "baseline:latest"
Warning: log-sammler-hpp5x (and 2 other pods): hostPath volumes
namespace/tag11-logs labeled (server dry run)
Warning: existing pods in namespace "tag11" violate the new PodSecurity enforce level
  "restricted:latest"
Warning: pod-lister-66f9bd75c5-vhcqz: allowPrivilegeEscalation != false, unrestricted
  capabilities, runAsNonRoot != true, seccompProfile
namespace/tag11 labeled (server dry run)

For new Pods, there are the modes warn and audit alongside enforce:

# on cp1
kubectl label namespace tag11 pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted
kubectl -n tag11 run probe --image=busybox:1.37 -- sleep 3600 2>&1 | cut -c1-88
kubectl -n tag11 delete pod probe --wait=false

Expected:

namespace/tag11 labeled
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != fal
pod/probe created
pod "probe" deleted from tag11 namespace

Why? warn reports to the caller, audit writes an annotation into the API server’s audit log – in the lab, RKE2’s audit log is off, so you see nothing of it there. Both let the Pod through. A namespace typically runs with warn for a few weeks before enforce follows.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. May the ServiceAccount tag11:app delete Pods in tag11? Target time: 15 s
  2. Show the PSA enforce label for all namespaces. Target time: 15 s
  3. Show how many Pods of the DaemonSet log-sammler should run and do run. Target time: 15 s

Solution

# on cp1
kubectl auth can-i delete pods -n tag11 --as=system:serviceaccount:tag11:app
kubectl get ns -L pod-security.kubernetes.io/enforce
kubectl -n tag11-logs get ds log-sammler \
  -o jsonpath='{.status.desiredNumberScheduled} {.status.numberReady}{"\n"}'

Expected (abridged):

no
NAME                 STATUS   AGE     ENFORCE
default              Active   150m
...
tag11                Active   6m48s
tag11-logs           Active   112s    privileged
tag11-streng         Active   87s     restricted
3 3

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl get nodes --as=system:serviceaccount:tag11:app -o name

Expected:

node/cp1
node/worker1
node/worker2
# on cp1
kubectl get pods -n tag11-logs -o name | wc -l
kubectl -n tag11-streng get deploy web

Expected:

3
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    1/1     1            1           2m

Challenge

A Job is supposed to list the ConfigMaps in tag11. Role and binding exist, yet the Job fails. Find the error without looking at the logs, and fix the file so that another apply does not bring it back.

# on cp1
cat > cm-job.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: cm-reader, namespace: tag11}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: cm-reader, namespace: tag11}
rules:
- apiGroups: [apps]
  resources: [configmaps]
  verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: cm-reader, namespace: tag11}
subjects:
- {kind: ServiceAccount, name: cm-reader, namespace: tag11}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: cm-reader}
---
apiVersion: batch/v1
kind: Job
metadata: {name: cm-reader, namespace: tag11}
spec:
  backoffLimit: 1
  template:
    spec:
      serviceAccountName: cm-reader
      restartPolicy: Never
      containers:
      - name: kubectl
        image: registry.k8s.io/kubectl:v1.37.2
        args: [get, configmaps, -n, tag11]
EOF
kubectl apply -f cm-job.yaml 2>&1 | grep -v Warning

Hint: Which API group do ConfigMaps belong to?

Solution

After about 30 seconds:

# on cp1
kubectl -n tag11 get job cm-reader
kubectl auth can-i list configmaps -n tag11 --as=system:serviceaccount:tag11:cm-reader
kubectl -n tag11 get role cm-reader -o jsonpath='{.rules}{"\n"}'
kubectl api-resources --api-group='' | grep -w configmaps

Expected:

NAME        STATUS   COMPLETIONS   DURATION   AGE
cm-reader   Failed   0/1           30s        30s
no
[{"apiGroups":["apps"],"resources":["configmaps"],"verbs":["get","list"]}]
configmaps               cm           v1           true         ConfigMap

ConfigMaps belong to the core group "" (APIVERSION v1 without a group), not to apps. RBAC checks group and resource together; the rule matches nothing. Put the fix into the file, otherwise the next apply restores the error:

# on cp1
sed -i 's/apiGroups: \[apps\]/apiGroups: [""]/' cm-job.yaml
kubectl -n tag11 delete job cm-reader
kubectl apply -f cm-job.yaml 2>&1 | grep -v Warning
kubectl -n tag11 wait job/cm-reader --for=condition=Complete
kubectl -n tag11 logs job/cm-reader

Expected:

job.batch "cm-reader" deleted from tag11 namespace
serviceaccount/cm-reader unchanged
role.rbac.authorization.k8s.io/cm-reader configured
rolebinding.rbac.authorization.k8s.io/cm-reader unchanged
job.batch/cm-reader created
job.batch/cm-reader condition met
NAME               DATA   AGE
kube-root-ca.crt   1      6m37s

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag11 tag11-logs tag11-streng
kubectl delete clusterrole,clusterrolebinding tag11-node-reader
kubectl get ns -L pod-security.kubernetes.io/enforce | grep tag11

Expected:

namespace "tag11" deleted
namespace "tag11-logs" deleted
namespace "tag11-streng" deleted
clusterrole.rbac.authorization.k8s.io "tag11-node-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "tag11-node-reader" deleted

Interview Note

  • How do you read a Forbidden message? User, verb, resource, API group, scope – each part points to a spot in the Role, the binding or the namespace.
  • Role or ClusterRole? Cluster-wide resources (nodes, PVs, namespaces) and access across all namespaces need a ClusterRole plus a ClusterRoleBinding.
  • Deployment exists, but no Pods? Admission (PSA, quota, webhook) – event on the ReplicaSet or DaemonSet.

Enterprise Note

  • Nobody works as system:masters day to day; personal access goes through OIDC with groups, so RBAC errors show up before the application finds them.
  • Set PSA labels via GitOps or a policy engine; new namespaces get at least baseline and warn=restricted.
  • Configure an audit policy with at least level: Metadata (RKE2: audit-policy-file; even with profile: cis the bundled policy logs nothing), otherwise audit has no effect.
  • Rules finer than PSA (image origin, labels, resources) are the job of a policy engine – the fault patterns for that are on Day 12.