Day 11: RBAC and Pod Security
Goal: You read a
Forbiddenmessage down to the last word, check permissions withauth can-iand find Pods that Pod Security Admission rejected. · Duration: approx. 60 min · Prerequisite: Day 3
Briefing
- An RBAC message names everything: who
(
User "system:serviceaccount:ns:name"), verb, resource, API group and scope (in the namespace …orat the cluster scope). kubectl auth can-i … --as=…checks permissions without starting the Pod.- Pod Security Admission (PSA) checks Pods. A Deployment is accepted, but its Pods are rejected – the event is on the ReplicaSet or DaemonSet.
- RKE2 sets
enforce: privilegedcluster-wide; restrictions come via namespace labels.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Binding in the wrong namespace | … in the namespace |
get rolebinding -A |
| Role instead of ClusterRole | at the cluster scope |
Cluster-wide resource? |
PSA baseline |
DaemonSet CURRENT 0 |
Event FailedCreate |
PSA restricted |
Deployment 0/1, no Pods |
Event on the ReplicaSet |
Drill 1: The RoleBinding in the wrong namespace
Break it: A Pod that lists Pods with
kubectl. The RoleBinding was copied from a template:
# on cp1
mkdir -p ~/debug/tag11 && cd ~/debug/tag11
kubectl create namespace tag11
cat > lister.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: app, namespace: tag11}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: pod-reader, namespace: tag11}
rules:
- apiGroups: [""]
resources: [pods]
verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: pod-reader, namespace: default}
subjects:
- {kind: ServiceAccount, name: app, namespace: tag11}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: pod-reader}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: pod-lister, namespace: tag11}
spec:
replicas: 1
selector: {matchLabels: {app: pod-lister}}
template:
metadata: {labels: {app: pod-lister}}
spec:
serviceAccountName: app
containers:
- name: kubectl
image: registry.k8s.io/kubectl:v1.37.2
args: [get, pods, -n, tag11, --watch]
EOF
kubectl apply -f lister.yamlSymptom (after about 20 seconds):
# on cp1
kubectl -n tag11 get pods
kubectl -n tag11 logs deploy/pod-lister --tail=1Expected (line wrapped):
NAME READY STATUS RESTARTS AGE
pod-lister-754f7856b7-qt7wh 1/1 Running 2 (12s ago) 20s
Error from server (Forbidden): pods is forbidden: User
"system:serviceaccount:tag11:app" cannot list resource "pods" in API group "" in
the namespace "tag11"
Investigate: Read the message: who? The
ServiceAccount app from tag11. What?
list pods, core group "". Where?
tag11. Now check the permissions for exactly that and look
for the bindings that name the account:
# on cp1
kubectl auth can-i list pods -n tag11 --as=system:serviceaccount:tag11:app
kubectl auth can-i list pods -n default --as=system:serviceaccount:tag11:app
kubectl get rolebinding -A -o custom-columns=NS:.metadata.namespace,\
NAME:.metadata.name,ROLE:.roleRef.name,SA:.subjects[*].name | grep -E '^NS|app' \
| tr -s ' 'Expected:
no
no - RBAC: role.rbac.authorization.k8s.io "pod-reader" not found
NS NAME ROLE SA
default pod-reader pod-reader app
Cause: The RoleBinding lives in default
and refers there to a Role pod-reader that only exists in
tag11. A RoleBinding only applies in its own namespace and
binds either a Role from that namespace or a ClusterRole. The second
can-i line even gives this away.
Fix:
# on cp1
kubectl -n default delete rolebinding pod-reader
kubectl -n tag11 create rolebinding pod-reader --role=pod-reader \
--serviceaccount=tag11:app
kubectl -n tag11 rollout restart deploy/pod-lister
kubectl -n tag11 rollout status deploy/pod-listerVerify:
# on cp1
kubectl auth can-i list pods -n tag11 --as=system:serviceaccount:tag11:app
kubectl -n tag11 logs deploy/pod-lister | head -2Expected:
yes
NAME READY STATUS RESTARTS AGE
pod-lister-66f9bd75c5-vhcqz 1/1 Running 0 2s
Why the restart? The container had exited with code 1 on every attempt and was in backoff (Day 5). Permissions take effect immediately, the backoff does not reset.
Save your progress? Sign up for free
Drill 2: Role instead of ClusterRole
Break it: The same ServiceAccount should read nodes:
# on cp1
cat > nodes.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: node-reader, namespace: tag11}
rules:
- apiGroups: [""]
resources: [nodes]
verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: node-reader, namespace: tag11}
subjects:
- {kind: ServiceAccount, name: app, namespace: tag11}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: node-reader}
EOF
kubectl apply -f nodes.yamlSymptom:
# on cp1
kubectl get nodes --as=system:serviceaccount:tag11:appExpected (line wrapped):
Error from server (Forbidden): nodes is forbidden: User
"system:serviceaccount:tag11:app" cannot list resource "nodes" in API group "" at
the cluster scope
Investigate:
# on cp1
kubectl auth can-i list nodes --as=system:serviceaccount:tag11:app
kubectl auth can-i list nodes -n tag11 --as=system:serviceaccount:tag11:app
kubectl api-resources | grep -E '^NAME|^nodes ' | tr -s ' 'Expected:
Warning: resource 'nodes' is not namespace scoped
no
Warning: resource 'nodes' is not namespace scoped
yes
NAME SHORTNAMES APIVERSION NAMESPACED KIND
nodes no v1 false Node
nodes metrics.k8s.io/v1beta1 false NodeMetrics
Cause: at the cluster scope – nodes do
not belong to any namespace (NAMESPACED false). A Role can
name them, but it only applies inside tag11, and there are
no nodes there. The yes with -n tag11 is a
trap: can-i checks the rule formally and only warns.
Fix:
# on cp1
kubectl delete -f nodes.yaml
kubectl create clusterrole tag11-node-reader --verb=get,list,watch --resource=nodes
kubectl create clusterrolebinding tag11-node-reader \
--clusterrole=tag11-node-reader --serviceaccount=tag11:appVerify:
# on cp1
kubectl get nodes --as=system:serviceaccount:tag11:appExpected:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane,etcd 148m v1.37.2+rke2r1
worker1 Ready <none> 145m v1.37.2+rke2r1
worker2 Ready <none> 144m v1.37.2+rke2r1
Save your progress? Sign up for free
Drill 3: All permissions at a glance
# on cp1
kubectl auth can-i --list -n tag11 --as=system:serviceaccount:tag11:app | head -6 \
| cut -c1-88
kubectl auth whoami | cut -c1-88Expected (abridged):
Resources Non-Resource URLs R
selfsubjectreviews.authentication.k8s.io [] [
selfsubjectaccessreviews.authorization.k8s.io [] [
selfsubjectrulesreviews.authorization.k8s.io [] [
nodes [] [
pods [] [
ATTRIBUTE VALUE
Username system:admin
Groups [system:masters system:authenticated
Extra: authentication.kubernetes.io/credential-id [X509SHA256=7d22b684b588198df48b975e
Why? --list shows the sum of all bindings for the
namespace – including those via ClusterRoleBindings.
auth whoami answers who you are right now:
the RKE2 kubeconfig is system:admin in
system:masters, and that group bypasses RBAC entirely. That
is why RBAC errors never show up when you test as admin.
Save your progress? Sign up for free
Drill 4: Pod Security baseline and the hostPath
Break it: A log collector as a DaemonSet in a
namespace with baseline:
# on cp1
kubectl create namespace tag11-logs
kubectl label namespace tag11-logs pod-security.kubernetes.io/enforce=baseline
cat > sammler.yaml <<'EOF'
apiVersion: apps/v1
kind: DaemonSet
metadata: {name: log-sammler, namespace: tag11-logs}
spec:
selector: {matchLabels: {app: log-sammler}}
template:
metadata: {labels: {app: log-sammler}}
spec:
containers:
- name: sammler
image: busybox:1.37
command: [sh, -c, 'tail -F /host-logs/syslog']
volumeMounts: [{name: logs, mountPath: /host-logs, readOnly: true}]
volumes:
- name: logs
hostPath: {path: /var/log}
EOF
kubectl apply -f sammler.yamlExpected:
Warning: would violate PodSecurity "baseline:latest": hostPath volumes (volume "logs")
daemonset.apps/log-sammler created
The warning already appears on apply – the DaemonSet
itself is created anyway.
Symptom:
# on cp1
kubectl -n tag11-logs get ds log-sammler | cut -c1-72
kubectl -n tag11-logs get podsExpected:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR
log-sammler 3 0 0 0 0 <none>
No resources found in tag11-logs namespace.
Investigate:
# on cp1
kubectl -n tag11-logs events --types=Warning | tail -1
kubectl get ns tag11-logs -L pod-security.kubernetes.io/enforceExpected (line wrapped):
1s (x2 over 3s) Warning FailedCreate DaemonSet/log-sammler (combined from
similar events): Error creating: pods "log-sammler-vcmsj" is forbidden: violates
PodSecurity "baseline:latest": hostPath volumes (volume "logs")
NAME STATUS AGE ENFORCE
tag11-logs Active 7s baseline
Cause: baseline forbids
hostPath. But a log collector needs exactly that. This is
not a bug in the Pod but a decision about the namespace: system-level
services get their own namespace with privileged.
Fix:
# on cp1
kubectl label --overwrite namespace tag11-logs \
pod-security.kubernetes.io/enforce=privilegedVerify:
# on cp1
kubectl -n tag11-logs rollout status ds/log-sammler
kubectl -n tag11-logs get ds log-sammler | cut -c1-72Expected:
daemon set "log-sammler" successfully rolled out
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR
log-sammler 3 3 3 3 3 <none>
The DaemonSet controller keeps retrying with backoff; depending on the number of failed attempts, this takes up to a minute. No restart is needed.
Talos: There,
baselineis the cluster-wide default – the same log collector fails without any label (Day 19).
Save your progress? Sign up for free
Drill 5: restricted – accepted, yet rejected
Break it:
# on cp1
kubectl create namespace tag11-streng
kubectl label namespace tag11-streng pod-security.kubernetes.io/enforce=restricted
kubectl -n tag11-streng create deployment web --image=nginxinc/nginx-unprivileged:1.29Expected (abridged):
namespace/tag11-streng created
namespace/tag11-streng labeled
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation !=
false (container "nginx-unprivileged" must set ...
deployment.apps/web created
Symptom:
# on cp1
kubectl -n tag11-streng get deploy web
kubectl -n tag11-streng get podsExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
web 0/1 0 0 4s
No resources found in tag11-streng namespace.
Investigate: As on Day 3 – no Pod, so go one level up:
# on cp1
kubectl -n tag11-streng events --types=Warning | tail -1 | fold -w 86Expected:
10s (x4 over 28s) Warning FailedCreate ReplicaSet/web-86c6bddd97 (combined fro
m similar events): Error creating: pods "web-86c6bddd97-hmhpn" is forbidden: violates
PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx-u
nprivileged" must set securityContext.allowPrivilegeEscalation=false), unrestricted ca
pabilities (container "nginx-unprivileged" must set securityContext.capabilities.drop=
["ALL"]), runAsNonRoot != true (pod or container "nginx-unprivileged" must set securit
yContext.runAsNonRoot=true), seccompProfile (pod or container "nginx-unprivileged" mus
t set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Cause: restricted requires four things
that hardly any manifest brings along on its own:
allowPrivilegeEscalation: false,
capabilities.drop: [ALL], runAsNonRoot: true
and a seccompProfile. The message lists each point
separately.
Fix:
# on cp1
kubectl -n tag11-streng patch deploy web -p '
spec:
template:
spec:
securityContext:
runAsNonRoot: true
seccompProfile: {type: RuntimeDefault}
containers:
- name: nginx-unprivileged
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: [ALL]}'
kubectl -n tag11-streng rollout status deploy/webVerify:
# on cp1
kubectl -n tag11-streng get podsExpected:
NAME READY STATUS RESTARTS AGE
web-6c88d4d554-xrcqn 1/1 Running 0 2s
Why does runAsNonRoot work here?
nginx-unprivileged sets the numeric user 101 in the image.
With busybox (root) you would also need
runAsUser.
Save your progress? Sign up for free
Drill 6: Warn first, then enforce
Before you tighten a namespace, you ask the API server what would happen – without changing anything:
# on cp1
kubectl label --dry-run=server --overwrite namespace tag11-logs \
pod-security.kubernetes.io/enforce=baseline
kubectl label --dry-run=server --overwrite namespace tag11 \
pod-security.kubernetes.io/enforce=restrictedExpected (line wrapped):
Warning: existing pods in namespace "tag11-logs" violate the new PodSecurity enforce
level "baseline:latest"
Warning: log-sammler-hpp5x (and 2 other pods): hostPath volumes
namespace/tag11-logs labeled (server dry run)
Warning: existing pods in namespace "tag11" violate the new PodSecurity enforce level
"restricted:latest"
Warning: pod-lister-66f9bd75c5-vhcqz: allowPrivilegeEscalation != false, unrestricted
capabilities, runAsNonRoot != true, seccompProfile
namespace/tag11 labeled (server dry run)
For new Pods, there are the modes warn and
audit alongside enforce:
# on cp1
kubectl label namespace tag11 pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/audit=restricted
kubectl -n tag11 run probe --image=busybox:1.37 -- sleep 3600 2>&1 | cut -c1-88
kubectl -n tag11 delete pod probe --wait=falseExpected:
namespace/tag11 labeled
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != fal
pod/probe created
pod "probe" deleted from tag11 namespace
Why? warn reports to the caller, audit
writes an annotation into the API server’s audit log – in the lab,
RKE2’s audit log is off, so you see nothing of it there. Both let the
Pod through. A namespace typically runs with warn for a few
weeks before enforce follows.
Save your progress? Sign up for free
Exercise: Speed Drills
- May the ServiceAccount
tag11:appdelete Pods intag11? Target time: 15 s - Show the PSA
enforcelabel for all namespaces. Target time: 15 s - Show how many Pods of the DaemonSet
log-sammlershould run and do run. Target time: 15 s
Solution
# on cp1
kubectl auth can-i delete pods -n tag11 --as=system:serviceaccount:tag11:app
kubectl get ns -L pod-security.kubernetes.io/enforce
kubectl -n tag11-logs get ds log-sammler \
-o jsonpath='{.status.desiredNumberScheduled} {.status.numberReady}{"\n"}'Expected (abridged):
no
NAME STATUS AGE ENFORCE
default Active 150m
...
tag11 Active 6m48s
tag11-logs Active 112s privileged
tag11-streng Active 87s restricted
3 3
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl get nodes --as=system:serviceaccount:tag11:app -o nameExpected:
node/cp1
node/worker1
node/worker2
# on cp1
kubectl get pods -n tag11-logs -o name | wc -l
kubectl -n tag11-streng get deploy webExpected:
3
NAME READY UP-TO-DATE AVAILABLE AGE
web 1/1 1 1 2m
Challenge
A Job is supposed to list the ConfigMaps in tag11. Role
and binding exist, yet the Job fails. Find the error without looking at
the logs, and fix the file so that another apply does not
bring it back.
# on cp1
cat > cm-job.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: cm-reader, namespace: tag11}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: cm-reader, namespace: tag11}
rules:
- apiGroups: [apps]
resources: [configmaps]
verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: cm-reader, namespace: tag11}
subjects:
- {kind: ServiceAccount, name: cm-reader, namespace: tag11}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: cm-reader}
---
apiVersion: batch/v1
kind: Job
metadata: {name: cm-reader, namespace: tag11}
spec:
backoffLimit: 1
template:
spec:
serviceAccountName: cm-reader
restartPolicy: Never
containers:
- name: kubectl
image: registry.k8s.io/kubectl:v1.37.2
args: [get, configmaps, -n, tag11]
EOF
kubectl apply -f cm-job.yaml 2>&1 | grep -v WarningHint: Which API group do ConfigMaps belong to?
Solution
After about 30 seconds:
# on cp1
kubectl -n tag11 get job cm-reader
kubectl auth can-i list configmaps -n tag11 --as=system:serviceaccount:tag11:cm-reader
kubectl -n tag11 get role cm-reader -o jsonpath='{.rules}{"\n"}'
kubectl api-resources --api-group='' | grep -w configmapsExpected:
NAME STATUS COMPLETIONS DURATION AGE
cm-reader Failed 0/1 30s 30s
no
[{"apiGroups":["apps"],"resources":["configmaps"],"verbs":["get","list"]}]
configmaps cm v1 true ConfigMap
ConfigMaps belong to the core group "" (APIVERSION
v1 without a group), not to apps. RBAC checks
group and resource together; the rule matches nothing. Put the fix into
the file, otherwise the next apply restores the error:
# on cp1
sed -i 's/apiGroups: \[apps\]/apiGroups: [""]/' cm-job.yaml
kubectl -n tag11 delete job cm-reader
kubectl apply -f cm-job.yaml 2>&1 | grep -v Warning
kubectl -n tag11 wait job/cm-reader --for=condition=Complete
kubectl -n tag11 logs job/cm-readerExpected:
job.batch "cm-reader" deleted from tag11 namespace
serviceaccount/cm-reader unchanged
role.rbac.authorization.k8s.io/cm-reader configured
rolebinding.rbac.authorization.k8s.io/cm-reader unchanged
job.batch/cm-reader created
job.batch/cm-reader condition met
NAME DATA AGE
kube-root-ca.crt 1 6m37s
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag11 tag11-logs tag11-streng
kubectl delete clusterrole,clusterrolebinding tag11-node-reader
kubectl get ns -L pod-security.kubernetes.io/enforce | grep tag11Expected:
namespace "tag11" deleted
namespace "tag11-logs" deleted
namespace "tag11-streng" deleted
clusterrole.rbac.authorization.k8s.io "tag11-node-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "tag11-node-reader" deleted
Interview Note
- How do you read a Forbidden message? User, verb, resource, API group, scope – each part points to a spot in the Role, the binding or the namespace.
- Role or ClusterRole? Cluster-wide resources (nodes, PVs, namespaces) and access across all namespaces need a ClusterRole plus a ClusterRoleBinding.
- Deployment exists, but no Pods? Admission (PSA, quota, webhook) – event on the ReplicaSet or DaemonSet.
Enterprise Note
- Nobody works as
system:mastersday to day; personal access goes through OIDC with groups, so RBAC errors show up before the application finds them. - Set PSA labels via GitOps or a policy engine; new namespaces get at
least
baselineandwarn=restricted. - Configure an audit policy with at least
level: Metadata(RKE2:audit-policy-file; even withprofile: cisthe bundled policy logs nothing), otherwiseaudithas no effect. - Rules finer than PSA (image origin, labels, resources) are the job of a policy engine – the fault patterns for that are on Day 12.