Day 14: Pod Authentication and Authorization
Goal: You read how Cilium turns labels into identities, allow HTTP calls per identity and use Mutual Authentication (SPIFFE/SPIRE) to require both sides to prove who they are cryptographically before a single packet gets through. · Duration: approx. 90 min · Prerequisite: Day 12
Briefing
- CKNE: Network Security and Policy – “Implementing Pod-level Authentication and Authorization”.
- Cilium does not filter by IP, but by identity: every combination of security-relevant labels (including namespace and ServiceAccount) gets a numeric ID that travels with the packet.
- Authorization: CiliumNetworkPolicy with L7 rules per identity
(method, path, header). Authentication:
authentication.mode: required– the path opens only after the agents complete an mTLS handshake with SPIFFE certificates. - Mutual Authentication is beta in Cilium 1.20; it does not encrypt anything.
tag14/frontend (SA frontend) ── GET /notes, /version ──┐
tag14/admin (SA admin) ── GET/POST /notes ───────┴─> notes-api :8080
cilium-agent <── mTLS (SVID spiffe://spiffe.cilium/identity/N) ──>
SPIRE server + agents (cilium-spire)
Drill 1: Labels become an identity
Three ServiceAccounts, notes-api with two replicas, two clients:
mkdir -p ~/ckne/tag14 && cd ~/ckne/tag14
k create namespace tag14
k config set-context --current --namespace=tag14
k create sa notes-api; k create sa frontend; k create sa admin
k create deployment notes-api --image=ghcr.io/thomaszachmann/notes-api:0.5.1 \
--replicas=2 --port=8080 $do > notes-api.yaml
k set env --local -f notes-api.yaml NOTES_API_KEY=lab-key-123 -o yaml \
> notes-api-env.yaml
k set serviceaccount --local -f notes-api-env.yaml notes-api -o yaml \
> notes-api-sa.yaml
k apply -f notes-api-sa.yaml
k expose deployment notes-api --port=80 --target-port=8080
k run frontend --image=curlimages/curl:8.17.0 \
--overrides='{"spec":{"serviceAccountName":"frontend"}}' --command -- sleep infinity
k run admin --image=curlimages/curl:8.17.0 \
--overrides='{"spec":{"serviceAccountName":"admin"}}' --command -- sleep infinity
k rollout status deployment notes-api
k wait --for=condition=Ready pod frontend admin
k get cepExpected (abridged):
deployment "notes-api" successfully rolled out
pod/frontend condition met
pod/admin condition met
NAME SECURITY IDENTITY ENDPOINT STATE IPV4 IPV6
admin 51657 ready 10.0.2.130
frontend 38567 ready 10.0.1.183
notes-api-97d957675-4jdq4 61704 ready 10.0.2.25
notes-api-97d957675-z7vg6 61704 ready 10.0.1.152
Both notes-api Pods share one identity, no matter which node they run on. What it consists of:
ID=$(k get cep -l app=notes-api -o jsonpath='{.items[0].status.identity.id}')
k get ciliumidentity $ID -o yaml | sed -n '/security-labels/,$p'
k get pod -l app=notes-api --show-labels | head -2Expected:
security-labels:
k8s:app: notes-api
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name: tag14
k8s:io.cilium.k8s.policy.cluster: default
k8s:io.cilium.k8s.policy.serviceaccount: notes-api
k8s:io.kubernetes.pod.namespace: tag14
NAME READY STATUS RESTARTS AGE LABELS
notes-api-97d957675-4jdq4 1/1 Running 0 24m app=notes-api,pod-temp...
Namespace, namespace labels and ServiceAccount are part of it,
pod-template-hash is not – Cilium filters out volatile
labels, otherwise every rollout would get new IDs.
Save your progress? Sign up for free
Drill 2: One label changes the identity
k label pod frontend team=blue
sleep 15; k get cep frontend
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg identity get \
$(k get cep frontend -o jsonpath='{.status.identity.id}') | grep -E 'ID|team'
k label pod frontend team-
sleep 15; k get cep frontendExpected (abridged):
pod/frontend labeled
NAME SECURITY IDENTITY ENDPOINT STATE IPV4 IPV6
frontend 6975 ready 10.0.1.183
ID LABELS
k8s:team=blue
pod/frontend unlabeled
NAME SECURITY IDENTITY ENDPOINT STATE IPV4 IPV6
frontend 38567 ready 10.0.1.183
In between, ENDPOINT STATE may briefly show
waiting-for-identity. What this means for you: anyone
allowed to set labels on Pods can switch their identity – so prefer
anchoring policies to the ServiceAccount, which RBAC protects.
Save your progress? Sign up for free
Drill 3: L7 authorization per identity
frontend may read, admin may also write –
but only with an API key in the header. The label
io.cilium.k8s.policy.serviceaccount selects by
ServiceAccount:
cat > cnp-l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: notes-api-l7
spec:
endpointSelector:
matchLabels:
app: notes-api
ingress:
- fromEndpoints:
- matchLabels:
io.cilium.k8s.policy.serviceaccount: frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: GET
path: /notes
- method: GET
path: /version
- fromEndpoints:
- matchLabels:
io.cilium.k8s.policy.serviceaccount: admin
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: GET
path: /notes
- method: POST
path: /notes
headers:
- "X-API-Key: lab-key-123"
EOF
k apply -f cnp-l7.yaml
k get cnp
for p in frontend admin; do
echo "== $p"
k exec $p -- curl -s -w ' %{http_code}\n' notes-api/version
k exec $p -- curl -s -o /dev/null -w 'GET /notes %{http_code}\n' notes-api/notes
k exec $p -- curl -s -w ' %{http_code}\n' -X POST notes-api/notes \
-H 'X-API-Key: lab-key-123' -d "{\"text\":\"von $p\"}"
done
k run fremd --image=curlimages/curl:8.17.0 --rm -i --restart=Never -- \
curl -s -m3 -w '%{http_code}\n' notes-api/versionExpected (abridged):
ciliumnetworkpolicy.cilium.io/notes-api-l7 created
NAME AGE VALID
notes-api-l7 5s True
== frontend
0.5.1
200
GET /notes 200
Access denied
403
== admin
Access denied
403
GET /notes 200
{"id":"f510a6a1-ef13-4a1c-83b6-c9edcfcc2805","text":"von admin"}
201
000
pod "fremd" deleted from tag14 namespace
Two kinds of no: 403 Access denied comes from the Envoy
proxy (L4 allowed, L7 not), 000 after a timeout means the
packet was already dropped at L3/L4 – fremd (ServiceAccount
default) is not in any rule.
Save your progress? Sign up for free
Drill 4: Proof with Hubble
k exec frontend -- curl -s -o /dev/null -X POST notes-api/notes \
-H 'X-API-Key: lab-key-123' -d '{}'
k exec admin -- curl -s -o /dev/null notes-api/notes
hubble observe -P --port-forward-port 0 --namespace tag14 --type l7 --last 4
hubble observe -P --port-forward-port 0 --namespace tag14 --verdict DROPPED --last 2Expected (abridged):
... tag14/frontend:60666 (ID:38567) -> tag14/notes-api-97d957675-4jdq4:8080 (ID:61704)
http-request DROPPED (HTTP/1.1 POST http://notes-api/notes)
... tag14/frontend:60666 (ID:38567) <- tag14/notes-api-97d957675-4jdq4:8080 (ID:61704)
http-response FORWARDED (HTTP/1.1 403 0ms (POST http://notes-api/notes))
... tag14/admin:34468 (ID:51657) -> tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
http-request FORWARDED (HTTP/1.1 GET http://notes-api/notes)
... tag14/admin:34468 (ID:51657) <- tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
http-response FORWARDED (HTTP/1.1 200 2ms (GET http://notes-api/notes))
... tag14/fremd:56230 (ID:64062) <> tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
... tag14/fremd:56230 (ID:64062) <> tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
Policy denied DROPPED (TCP Flags: SYN)
Every line names the identity of both sides. So far, however, this is only a claim: the source identity sits in the tunnel header – anyone who controls a node or forges packets can fake it. Drill 6 closes that gap.
Save your progress? Sign up for free
Drill 5: Install SPIRE
Cilium ships a SPIRE server and SPIRE agents as part of its Helm
chart. The server stores its data in a PVC (default StorageClass
local-path).
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set authentication.enabled=true \
--set authentication.mutual.spire.enabled=true \
--set authentication.mutual.spire.install.enabled=true \
--set authentication.mutual.spire.install.server.securityContext.runAsUser=0 \
--set authentication.mutual.spire.install.server.securityContext.runAsGroup=0 \
--set authentication.mutual.spire.install.agent.securityContext.runAsUser=0 \
--set authentication.mutual.spire.install.agent.securityContext.runAsGroup=0
k -n kube-system rollout restart deployment/cilium-operator
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
k -n cilium-spire rollout status statefulset spire-server
k -n cilium-spire rollout status ds spire-agent
k -n cilium-spire get pods,pvc
cilium config view | grep -E '^mesh-auth-(enabled|mutual-enabled|spiffe-trust-domain)'Expected (abridged):
daemon set "cilium" successfully rolled out
statefulset rolling update complete 1 pods at revision spire-server-...
daemon set "spire-agent" successfully rolled out
NAME READY STATUS RESTARTS AGE
pod/spire-agent-2pbh8 1/1 Running 0 16s
pod/spire-agent-bt5rc 1/1 Running 0 16s
pod/spire-agent-bwjtr 1/1 Running 0 17s
pod/spire-server-0 2/2 Running 0 27s
NAME STATUS ... STORAGECLASS
persistentvolumeclaim/spire-data-spire-server-0 Bound ... local-path
mesh-auth-enabled true
mesh-auth-mutual-enabled true
mesh-auth-spiffe-trust-domain spiffe.cilium
Warning: The four
securityContextlines are a lab finding. Without them, the SPIRE server 1.15.5 from chart 1.20.5 does not start: the image runs as UID 1000 and is not allowed to create its admin socket in the hostPath/var/run/spire-server/sockets(root, 755). The server only logs “Starting Server APIs” followed immediately by “Stopping Server APIs”, then hangs and is restarted after the liveness probe; the agents wait in the init container “Waiting for spire server to be reachable”.
The SPIRE server now has three agents (one per node, attested via projected service account token) and one entry per Cilium identity, created by the operator. If identities existed before SPIRE, restarting the operator helps fill them in:
k -n kube-system rollout restart deploy cilium-operator
k -n kube-system rollout status deploy cilium-operator
S="kubectl -n cilium-spire exec spire-server-0 -c spire-server -- \
/opt/spire/bin/spire-server"
$S healthcheck
$S agent list | grep 'Attestation type'
$S entry show | grep -c 'identity/'
$S entry show -spiffeID spiffe://spiffe.cilium/identity/$IDExpected (abridged):
Server is healthy.
Attestation type : k8s_psat
Attestation type : k8s_psat
Attestation type : k8s_psat
112
Found 1 entry
Entry ID : 6d6f3e03-bcd8-4bc4-b961-c3dc50a48175
SPIFFE ID : spiffe://spiffe.cilium/identity/61704
Parent ID : spiffe://spiffe.cilium/cilium-operator
Selector : cilium:mutual-auth
The SPIFFE ID encodes the Cilium identity. The agents fetch the matching certificates (SVIDs) and use them to prove their identity to each other.
Save your progress? Sign up for free
Drill 6: Enforce authentication
One field per rule – authentication.mode: required:
k explain cnp.spec.ingress.authentication.mode | sed -n '/ENUM/,/DESCRIPTION/p'
AUTH=' - authentication:\n mode: required\n fromEndpoints:'
sed "s/^ - fromEndpoints:/$AUTH/" cnp-l7.yaml > cnp-auth.yaml
sed -n '/ingress:/,/toPorts/p' cnp-auth.yaml
k apply -f cnp-auth.yaml
for p in frontend admin; do
k exec $p -- curl -s -m5 -w ' %{http_code}\n' notes-api/version
done
hubble observe -P --port-forward-port 0 --namespace tag14 --verdict DROPPED \
--last 5 | grep -i auth
CP=$(k -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=worker1 \
-o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth listExpected (abridged):
ENUM:
disabled
required
test-always-fail
ingress:
- authentication:
mode: required
fromEndpoints:
- matchLabels:
io.cilium.k8s.policy.serviceaccount: frontend
toPorts:
ciliumnetworkpolicy.cilium.io/notes-api-l7 configured
0.5.1
200
Access denied
403
... tag14/frontend:50714 (ID:38567) <> tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
Authentication required DROPPED (TCP Flags: SYN)
... tag14/admin:42274 (ID:51657) <> tag14/notes-api-97d957675-4jdq4:8080 (ID:61704)
Authentication required DROPPED (TCP Flags: SYN)
SRC IDENTITY DST IDENTITY REMOTE NODE ID AUTH TYPE EXPIRATION
61704 51657 47089 spire 2026-11-19 01:44:45 +0000 UTC
61704 38567 0 spire 2026-11-19 01:44:44 +0000 UTC
How it works: the datapath drops the first SYN (“Authentication
required”) and triggers the handshake. The agents on both nodes verify
the SVIDs of both identities via mTLS (port 4250) and enter the pair
into the map cilium_auth_map – valid until the certificate
expires. The TCP retransmission of the SYN gets through; to the client
this looks like one second of latency. REMOTE NODE ID 0
means: both on the same node.
Counter-check with the test mode, which makes every handshake fail:
sed 's/mode: required/mode: test-always-fail/' cnp-auth.yaml | k apply -f -
k exec frontend -- curl -s -m5 -w ' %{http_code}\n' notes-api/version
hubble observe -P --port-forward-port 0 --namespace tag14 --verdict DROPPED --last 1
k apply -f cnp-auth.yaml
k exec frontend -- curl -s -m5 -w ' %{http_code}\n' notes-api/versionExpected (abridged):
ciliumnetworkpolicy.cilium.io/notes-api-l7 configured
000
command terminated with exit code 28
... tag14/frontend:<Port> (ID:38567) <> tag14/notes-api-…:8080 (ID:61704)
Authentication required DROPPED (TCP Flags: SYN)
ciliumnetworkpolicy.cilium.io/notes-api-l7 configured
0.5.1
200
Save your progress? Sign up for free
Drill 7: Authenticated does not mean encrypted
The handshake runs alongside the connection (“out-of-band”); the
payload itself stays untouched. Capture on the tunnel on
worker2 while admin calls the Pod on
worker1 directly:
P=$(k get pod -l app=notes-api --field-selector spec.nodeName=worker1 \
-o jsonpath='{.items[0].status.podIP}')
ssh [email protected] \
"sudo timeout 8 tcpdump -i eth0 -A -n udp port 8472 | grep -m1 'GET /version'" &
sleep 2
for i in 1 2 3; do k exec admin -- curl -s -m3 -o /dev/null http://$P:8080/version; done
waitExpected (abridged):
IP 10.0.2.130.43106 > 10.0.1.152.8080: Flags [P.], seq 1:87, ack 1, win 507, options
[nop,nop,TS val 3357360675 ecr 3655487247], length 86: HTTP: GET /version HTTP/1.1
The HTTP request sits in plain text inside the VXLAN packet (UDP 8472). Mutual Authentication answers “who is talking?”, not “who is listening?”. That is why the Cilium docs additionally require encryption (WireGuard or IPsec, Day 13). Coupling the two is still marked “TODO” on the roadmap.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all endpoints in
tag14with their identity ID in two columns. Target time: 45 s - Show the security labels of the identity of
admin. Target time: 60 s - Additionally allow
adminto callGET /versionwithout touching other rules. Target time: 2 min - Show the authentication entries on
worker2. Target time: 60 s
Solution
k get cep -o custom-columns=NAME:.metadata.name,ID:.status.identity.id
k get ciliumidentity $(k get cep admin -o jsonpath='{.status.identity.id}') \
-o jsonpath='{.security-labels}'; echo
k get cnp notes-api-l7 -o jsonpath='{.spec.ingress[1].toPorts[0].rules.http}'; echo
k patch cnp notes-api-l7 --type=json -p '[{"op":"add",
"path":"/spec/ingress/1/toPorts/0/rules/http/-",
"value":{"method":"GET","path":"/version"}}]'
k exec admin -- curl -s -w ' %{http_code}\n' notes-api/version
CP=$(k -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=worker2 \
-o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth listExpected (abridged):
NAME ID
admin 51657
frontend 38567
notes-api-97d957675-4jdq4 61704
notes-api-97d957675-z7vg6 61704
{"k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name":"tag14",...,
"k8s:io.cilium.k8s.policy.serviceaccount":"admin",...,"k8s:run":"admin"}
[{"method":"GET","path":"/notes"},{"headers":["X-API-Key: lab-key-123"],...}]
ciliumnetworkpolicy.cilium.io/notes-api-l7 patched
0.5.1
200
SRC IDENTITY DST IDENTITY REMOTE NODE ID AUTH TYPE EXPIRATION
61704 38567 52624 spire 2026-11-19 01:44:44 +0000 UTC
61704 51657 0 spire 2026-11-19 01:44:49 +0000 UTC
Save your progress? Sign up for free
Checkpoint
k get cnp notes-api-l7 -o jsonpath='{.spec.ingress[*].authentication.mode}{"\n"}'Expected: required required
k exec frontend -- curl -s -w ' %{http_code}\n' -X POST notes-api/notes -d '{}'Expected: Access denied, followed by
403.
k -n cilium-spire exec spire-server-0 -c spire-server -- \
/opt/spire/bin/spire-server healthcheckExpected: Server is healthy.
hubble observe -P --port-forward-port 0 --namespace tag14 --verdict DROPPED \
--last 20 | grep -c 'Authentication required'Expected: a number greater than 0.
Challenge
Start a second Pod frontend2 with the same labels and
the same ServiceAccount as frontend on
worker1. Does it need its own handshake?
Hint: Compare the identity and
cilium-dbg bpf auth list before and after its first
call.
Solution
k run frontend2 -l run=frontend --image=curlimages/curl:8.17.0 \
--overrides='{"spec":{"serviceAccountName":"frontend","nodeName":"worker1"}}' \
--command -- sleep infinity
k wait --for=condition=Ready pod/frontend2
k get cep frontend frontend2
CP=$(k -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=worker1 \
-o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth list
k exec frontend2 -- curl -s -m5 -w ' %{http_code}\n' notes-api/version
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth listExpected:
pod/frontend2 created
pod/frontend2 condition met
NAME SECURITY IDENTITY ENDPOINT STATE IPV4 IPV6
frontend 38567 ready 10.0.1.183
frontend2 38567 ready 10.0.1.79
SRC IDENTITY DST IDENTITY REMOTE NODE ID AUTH TYPE EXPIRATION
61704 51657 47089 spire 2026-11-19 01:44:45 +0000 UTC
61704 38567 0 spire 2026-11-19 01:44:44 +0000 UTC
0.5.1
200
SRC IDENTITY DST IDENTITY REMOTE NODE ID AUTH TYPE EXPIRATION
61704 51657 47089 spire 2026-11-19 01:44:45 +0000 UTC
61704 38567 0 spire 2026-11-19 01:44:44 +0000 UTC
No new entry, no “Authentication required” drop: authentication happens per identity pair, not per Pod or connection (roadmap: “Per-connection handshake – TODO”). Whoever carries the labels of an authorized Pod inherits its clearance – another reason to protect labels and ServiceAccounts with RBAC and admission control.
Save your progress? Sign up for free
Cleanup
Mutual Authentication is beta and occupies a SPIRE agent on every
node – back to the state before Day 14. The backup from Drill 5 contains
no authentication values; without
--reuse-values, the chart defaults apply again (both off),
and Helm removes server, agents and RBAC:
k delete namespace tag14
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart deployment/cilium-operator
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
k delete namespace cilium-spire --ignore-not-found
cilium config view | grep -E '^mesh-auth-enabled'
k config set-context --current --namespace=defaultExpected (abridged): GLEICH,
mesh-auth-enabled false; the namespace
cilium-spire including its PVC is gone.
Warning: Do not reset via
--reuse-values --set authentication=null– in the lab test, the chart then keeps rendering with Mutual Authentication and SPIRE.
Exam Tip
- Identity = labels + namespace + ServiceAccount. For “only Pods with
ServiceAccount X”, match
io.cilium.k8s.policy.serviceaccount: XinfromEndpoints. - L7 rules live under
toPorts[].rules.http(method, path as regex, header); an L7 no is a403, an L3/L4 no is a timeout – both visible inhubble observe. authentication.mode: requiredsits at the level offromEndpointsin the ingress rule; drop reason “Authentication required”. Docs: docs.cilium.io “Mutual Authentication (Beta)”.
Enterprise Note
- Cilium Mutual Authentication is beta: no per-connection handshake, cannot be combined with ClusterMesh, tested only with SPIRE. For compliance-grade mTLS today, rather use a mesh (Istio Ambient, Linkerd) or Cilium’s ztunnel integration (also beta).
- Always combine Mutual Authentication with WireGuard or IPsec – it does not replace encryption.
- SPIRE in production: a dedicated HA server with an external database, an upstream CA from Vault or your PKI, monitoring of SVID issuance; an outage means new identity pairs no longer get through.
- Labels are security attributes: use an admission policy (Kyverno,
VAP) to prevent teams from using other teams’
applabels or ServiceAccounts. - The identity labels can be restricted with the Helm value
labels– fewer identities, less policy load in large clusters.