Skip to content
Hands-on IT IT Drills

Day 14: Pod Authentication and Authorization

Goal: You read how Cilium turns labels into identities, allow HTTP calls per identity and use Mutual Authentication (SPIFFE/SPIRE) to require both sides to prove who they are cryptographically before a single packet gets through. · Duration: approx. 90 min · Prerequisite: Day 12

Briefing

  • CKNE: Network Security and Policy – “Implementing Pod-level Authentication and Authorization”.
  • Cilium does not filter by IP, but by identity: every combination of security-relevant labels (including namespace and ServiceAccount) gets a numeric ID that travels with the packet.
  • Authorization: CiliumNetworkPolicy with L7 rules per identity (method, path, header). Authentication: authentication.mode: required – the path opens only after the agents complete an mTLS handshake with SPIFFE certificates.
  • Mutual Authentication is beta in Cilium 1.20; it does not encrypt anything.
 tag14/frontend (SA frontend) ── GET /notes, /version ──┐
 tag14/admin    (SA admin)    ── GET/POST /notes ───────┴─> notes-api :8080
                 cilium-agent <── mTLS (SVID spiffe://spiffe.cilium/identity/N) ──>
                                         SPIRE server + agents (cilium-spire)

Drill 1: Labels become an identity

Three ServiceAccounts, notes-api with two replicas, two clients:

mkdir -p ~/ckne/tag14 && cd ~/ckne/tag14
k create namespace tag14
k config set-context --current --namespace=tag14
k create sa notes-api; k create sa frontend; k create sa admin
k create deployment notes-api --image=ghcr.io/thomaszachmann/notes-api:0.5.1 \
  --replicas=2 --port=8080 $do > notes-api.yaml
k set env --local -f notes-api.yaml NOTES_API_KEY=lab-key-123 -o yaml \
  > notes-api-env.yaml
k set serviceaccount --local -f notes-api-env.yaml notes-api -o yaml \
  > notes-api-sa.yaml
k apply -f notes-api-sa.yaml
k expose deployment notes-api --port=80 --target-port=8080
k run frontend --image=curlimages/curl:8.17.0 \
  --overrides='{"spec":{"serviceAccountName":"frontend"}}' --command -- sleep infinity
k run admin --image=curlimages/curl:8.17.0 \
  --overrides='{"spec":{"serviceAccountName":"admin"}}' --command -- sleep infinity
k rollout status deployment notes-api
k wait --for=condition=Ready pod frontend admin
k get cep

Expected (abridged):

deployment "notes-api" successfully rolled out
pod/frontend condition met
pod/admin condition met
NAME                        SECURITY IDENTITY   ENDPOINT STATE   IPV4         IPV6
admin                       51657               ready            10.0.2.130
frontend                    38567               ready            10.0.1.183
notes-api-97d957675-4jdq4   61704               ready            10.0.2.25
notes-api-97d957675-z7vg6   61704               ready            10.0.1.152

Both notes-api Pods share one identity, no matter which node they run on. What it consists of:

ID=$(k get cep -l app=notes-api -o jsonpath='{.items[0].status.identity.id}')
k get ciliumidentity $ID -o yaml | sed -n '/security-labels/,$p'
k get pod -l app=notes-api --show-labels | head -2

Expected:

security-labels:
  k8s:app: notes-api
  k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name: tag14
  k8s:io.cilium.k8s.policy.cluster: default
  k8s:io.cilium.k8s.policy.serviceaccount: notes-api
  k8s:io.kubernetes.pod.namespace: tag14
NAME                        READY   STATUS    RESTARTS   AGE   LABELS
notes-api-97d957675-4jdq4   1/1     Running   0          24m   app=notes-api,pod-temp...

Namespace, namespace labels and ServiceAccount are part of it, pod-template-hash is not – Cilium filters out volatile labels, otherwise every rollout would get new IDs.

Save your progress? Sign up for free

Drill 2: One label changes the identity

k label pod frontend team=blue
sleep 15; k get cep frontend
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg identity get \
  $(k get cep frontend -o jsonpath='{.status.identity.id}') | grep -E 'ID|team'
k label pod frontend team-
sleep 15; k get cep frontend

Expected (abridged):

pod/frontend labeled
NAME       SECURITY IDENTITY   ENDPOINT STATE   IPV4         IPV6
frontend   6975                ready            10.0.1.183
ID     LABELS
       k8s:team=blue
pod/frontend unlabeled
NAME       SECURITY IDENTITY   ENDPOINT STATE   IPV4         IPV6
frontend   38567               ready            10.0.1.183

In between, ENDPOINT STATE may briefly show waiting-for-identity. What this means for you: anyone allowed to set labels on Pods can switch their identity – so prefer anchoring policies to the ServiceAccount, which RBAC protects.

Save your progress? Sign up for free

Drill 3: L7 authorization per identity

frontend may read, admin may also write – but only with an API key in the header. The label io.cilium.k8s.policy.serviceaccount selects by ServiceAccount:

cat > cnp-l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: notes-api-l7
spec:
  endpointSelector:
    matchLabels:
      app: notes-api
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: /notes
        - method: GET
          path: /version
  - fromEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: admin
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: /notes
        - method: POST
          path: /notes
          headers:
          - "X-API-Key: lab-key-123"
EOF
k apply -f cnp-l7.yaml
k get cnp
for p in frontend admin; do
  echo "== $p"
  k exec $p -- curl -s -w ' %{http_code}\n' notes-api/version
  k exec $p -- curl -s -o /dev/null -w 'GET /notes %{http_code}\n' notes-api/notes
  k exec $p -- curl -s -w ' %{http_code}\n' -X POST notes-api/notes \
    -H 'X-API-Key: lab-key-123' -d "{\"text\":\"von $p\"}"
done
k run fremd --image=curlimages/curl:8.17.0 --rm -i --restart=Never -- \
  curl -s -m3 -w '%{http_code}\n' notes-api/version

Expected (abridged):

ciliumnetworkpolicy.cilium.io/notes-api-l7 created
NAME           AGE   VALID
notes-api-l7   5s    True
== frontend
0.5.1
 200
GET /notes 200
Access denied
 403
== admin
Access denied
 403
GET /notes 200
{"id":"f510a6a1-ef13-4a1c-83b6-c9edcfcc2805","text":"von admin"}
 201
000
pod "fremd" deleted from tag14 namespace

Two kinds of no: 403 Access denied comes from the Envoy proxy (L4 allowed, L7 not), 000 after a timeout means the packet was already dropped at L3/L4 – fremd (ServiceAccount default) is not in any rule.

Save your progress? Sign up for free

Drill 4: Proof with Hubble

k exec frontend -- curl -s -o /dev/null -X POST notes-api/notes \
  -H 'X-API-Key: lab-key-123' -d '{}'
k exec admin -- curl -s -o /dev/null notes-api/notes
hubble observe -P --port-forward-port 0 --namespace tag14 --type l7 --last 4
hubble observe -P --port-forward-port 0 --namespace tag14 --verdict DROPPED --last 2

Expected (abridged):

... tag14/frontend:60666 (ID:38567) -> tag14/notes-api-97d957675-4jdq4:8080 (ID:61704)
http-request DROPPED (HTTP/1.1 POST http://notes-api/notes)
... tag14/frontend:60666 (ID:38567) <- tag14/notes-api-97d957675-4jdq4:8080 (ID:61704)
http-response FORWARDED (HTTP/1.1 403 0ms (POST http://notes-api/notes))
... tag14/admin:34468 (ID:51657) -> tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
http-request FORWARDED (HTTP/1.1 GET http://notes-api/notes)
... tag14/admin:34468 (ID:51657) <- tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
http-response FORWARDED (HTTP/1.1 200 2ms (GET http://notes-api/notes))
... tag14/fremd:56230 (ID:64062) <> tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
... tag14/fremd:56230 (ID:64062) <> tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
Policy denied DROPPED (TCP Flags: SYN)

Every line names the identity of both sides. So far, however, this is only a claim: the source identity sits in the tunnel header – anyone who controls a node or forges packets can fake it. Drill 6 closes that gap.

Save your progress? Sign up for free

Drill 5: Install SPIRE

Cilium ships a SPIRE server and SPIRE agents as part of its Helm chart. The server stores its data in a PVC (default StorageClass local-path).

helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set authentication.enabled=true \
  --set authentication.mutual.spire.enabled=true \
  --set authentication.mutual.spire.install.enabled=true \
  --set authentication.mutual.spire.install.server.securityContext.runAsUser=0 \
  --set authentication.mutual.spire.install.server.securityContext.runAsGroup=0 \
  --set authentication.mutual.spire.install.agent.securityContext.runAsUser=0 \
  --set authentication.mutual.spire.install.agent.securityContext.runAsGroup=0
k -n kube-system rollout restart deployment/cilium-operator
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
k -n cilium-spire rollout status statefulset spire-server
k -n cilium-spire rollout status ds spire-agent
k -n cilium-spire get pods,pvc
cilium config view | grep -E '^mesh-auth-(enabled|mutual-enabled|spiffe-trust-domain)'

Expected (abridged):

daemon set "cilium" successfully rolled out
statefulset rolling update complete 1 pods at revision spire-server-...
daemon set "spire-agent" successfully rolled out
NAME                    READY   STATUS    RESTARTS   AGE
pod/spire-agent-2pbh8   1/1     Running   0          16s
pod/spire-agent-bt5rc   1/1     Running   0          16s
pod/spire-agent-bwjtr   1/1     Running   0          17s
pod/spire-server-0      2/2     Running   0          27s
NAME                                              STATUS   ...   STORAGECLASS
persistentvolumeclaim/spire-data-spire-server-0   Bound    ...   local-path
mesh-auth-enabled                                 true
mesh-auth-mutual-enabled                          true
mesh-auth-spiffe-trust-domain                     spiffe.cilium

Warning: The four securityContext lines are a lab finding. Without them, the SPIRE server 1.15.5 from chart 1.20.5 does not start: the image runs as UID 1000 and is not allowed to create its admin socket in the hostPath /var/run/spire-server/sockets (root, 755). The server only logs “Starting Server APIs” followed immediately by “Stopping Server APIs”, then hangs and is restarted after the liveness probe; the agents wait in the init container “Waiting for spire server to be reachable”.

The SPIRE server now has three agents (one per node, attested via projected service account token) and one entry per Cilium identity, created by the operator. If identities existed before SPIRE, restarting the operator helps fill them in:

k -n kube-system rollout restart deploy cilium-operator
k -n kube-system rollout status deploy cilium-operator
S="kubectl -n cilium-spire exec spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server"
$S healthcheck
$S agent list | grep 'Attestation type'
$S entry show | grep -c 'identity/'
$S entry show -spiffeID spiffe://spiffe.cilium/identity/$ID

Expected (abridged):

Server is healthy.
Attestation type  : k8s_psat
Attestation type  : k8s_psat
Attestation type  : k8s_psat
112
Found 1 entry
Entry ID                : 6d6f3e03-bcd8-4bc4-b961-c3dc50a48175
SPIFFE ID               : spiffe://spiffe.cilium/identity/61704
Parent ID               : spiffe://spiffe.cilium/cilium-operator
Selector                : cilium:mutual-auth

The SPIFFE ID encodes the Cilium identity. The agents fetch the matching certificates (SVIDs) and use them to prove their identity to each other.

Save your progress? Sign up for free

Drill 6: Enforce authentication

One field per rule – authentication.mode: required:

k explain cnp.spec.ingress.authentication.mode | sed -n '/ENUM/,/DESCRIPTION/p'
AUTH='  - authentication:\n      mode: required\n    fromEndpoints:'
sed "s/^  - fromEndpoints:/$AUTH/" cnp-l7.yaml > cnp-auth.yaml
sed -n '/ingress:/,/toPorts/p' cnp-auth.yaml
k apply -f cnp-auth.yaml
for p in frontend admin; do
  k exec $p -- curl -s -m5 -w ' %{http_code}\n' notes-api/version
done
hubble observe -P --port-forward-port 0 --namespace tag14 --verdict DROPPED \
  --last 5 | grep -i auth
CP=$(k -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=worker1 \
  -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth list

Expected (abridged):

ENUM:
    disabled
    required
    test-always-fail

  ingress:
  - authentication:
      mode: required
    fromEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: frontend
    toPorts:
ciliumnetworkpolicy.cilium.io/notes-api-l7 configured
0.5.1
 200
Access denied
 403
... tag14/frontend:50714 (ID:38567) <> tag14/notes-api-97d957675-z7vg6:8080 (ID:61704)
Authentication required DROPPED (TCP Flags: SYN)
... tag14/admin:42274 (ID:51657) <> tag14/notes-api-97d957675-4jdq4:8080 (ID:61704)
Authentication required DROPPED (TCP Flags: SYN)
SRC IDENTITY   DST IDENTITY   REMOTE NODE ID   AUTH TYPE   EXPIRATION
61704          51657          47089            spire       2026-11-19 01:44:45 +0000 UTC
61704          38567          0                spire       2026-11-19 01:44:44 +0000 UTC

How it works: the datapath drops the first SYN (“Authentication required”) and triggers the handshake. The agents on both nodes verify the SVIDs of both identities via mTLS (port 4250) and enter the pair into the map cilium_auth_map – valid until the certificate expires. The TCP retransmission of the SYN gets through; to the client this looks like one second of latency. REMOTE NODE ID 0 means: both on the same node.

Counter-check with the test mode, which makes every handshake fail:

sed 's/mode: required/mode: test-always-fail/' cnp-auth.yaml | k apply -f -
k exec frontend -- curl -s -m5 -w ' %{http_code}\n' notes-api/version
hubble observe -P --port-forward-port 0 --namespace tag14 --verdict DROPPED --last 1
k apply -f cnp-auth.yaml
k exec frontend -- curl -s -m5 -w ' %{http_code}\n' notes-api/version

Expected (abridged):

ciliumnetworkpolicy.cilium.io/notes-api-l7 configured
 000
command terminated with exit code 28
... tag14/frontend:<Port> (ID:38567) <> tag14/notes-api-…:8080 (ID:61704)
Authentication required DROPPED (TCP Flags: SYN)
ciliumnetworkpolicy.cilium.io/notes-api-l7 configured
0.5.1
 200

Save your progress? Sign up for free

Drill 7: Authenticated does not mean encrypted

The handshake runs alongside the connection (“out-of-band”); the payload itself stays untouched. Capture on the tunnel on worker2 while admin calls the Pod on worker1 directly:

P=$(k get pod -l app=notes-api --field-selector spec.nodeName=worker1 \
  -o jsonpath='{.items[0].status.podIP}')
ssh [email protected] \
  "sudo timeout 8 tcpdump -i eth0 -A -n udp port 8472 | grep -m1 'GET /version'" &
sleep 2
for i in 1 2 3; do k exec admin -- curl -s -m3 -o /dev/null http://$P:8080/version; done
wait

Expected (abridged):

IP 10.0.2.130.43106 > 10.0.1.152.8080: Flags [P.], seq 1:87, ack 1, win 507, options
[nop,nop,TS val 3357360675 ecr 3655487247], length 86: HTTP: GET /version HTTP/1.1

The HTTP request sits in plain text inside the VXLAN packet (UDP 8472). Mutual Authentication answers “who is talking?”, not “who is listening?”. That is why the Cilium docs additionally require encryption (WireGuard or IPsec, Day 13). Coupling the two is still marked “TODO” on the roadmap.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all endpoints in tag14 with their identity ID in two columns. Target time: 45 s
  2. Show the security labels of the identity of admin. Target time: 60 s
  3. Additionally allow admin to call GET /version without touching other rules. Target time: 2 min
  4. Show the authentication entries on worker2. Target time: 60 s

Solution

k get cep -o custom-columns=NAME:.metadata.name,ID:.status.identity.id
k get ciliumidentity $(k get cep admin -o jsonpath='{.status.identity.id}') \
  -o jsonpath='{.security-labels}'; echo
k get cnp notes-api-l7 -o jsonpath='{.spec.ingress[1].toPorts[0].rules.http}'; echo
k patch cnp notes-api-l7 --type=json -p '[{"op":"add",
 "path":"/spec/ingress/1/toPorts/0/rules/http/-",
 "value":{"method":"GET","path":"/version"}}]'
k exec admin -- curl -s -w ' %{http_code}\n' notes-api/version
CP=$(k -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=worker2 \
  -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth list

Expected (abridged):

NAME                        ID
admin                       51657
frontend                    38567
notes-api-97d957675-4jdq4   61704
notes-api-97d957675-z7vg6   61704
{"k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name":"tag14",...,
"k8s:io.cilium.k8s.policy.serviceaccount":"admin",...,"k8s:run":"admin"}
[{"method":"GET","path":"/notes"},{"headers":["X-API-Key: lab-key-123"],...}]
ciliumnetworkpolicy.cilium.io/notes-api-l7 patched
0.5.1
 200
SRC IDENTITY   DST IDENTITY   REMOTE NODE ID   AUTH TYPE   EXPIRATION
61704          38567          52624            spire       2026-11-19 01:44:44 +0000 UTC
61704          51657          0                spire       2026-11-19 01:44:49 +0000 UTC

Save your progress? Sign up for free

Checkpoint

k get cnp notes-api-l7 -o jsonpath='{.spec.ingress[*].authentication.mode}{"\n"}'

Expected: required required

k exec frontend -- curl -s -w ' %{http_code}\n' -X POST notes-api/notes -d '{}'

Expected: Access denied, followed by 403.

k -n cilium-spire exec spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server healthcheck

Expected: Server is healthy.

hubble observe -P --port-forward-port 0 --namespace tag14 --verdict DROPPED \
  --last 20 | grep -c 'Authentication required'

Expected: a number greater than 0.

Challenge

Start a second Pod frontend2 with the same labels and the same ServiceAccount as frontend on worker1. Does it need its own handshake?

Hint: Compare the identity and cilium-dbg bpf auth list before and after its first call.

Solution

k run frontend2 -l run=frontend --image=curlimages/curl:8.17.0 \
  --overrides='{"spec":{"serviceAccountName":"frontend","nodeName":"worker1"}}' \
  --command -- sleep infinity
k wait --for=condition=Ready pod/frontend2
k get cep frontend frontend2
CP=$(k -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=worker1 \
  -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth list
k exec frontend2 -- curl -s -m5 -w ' %{http_code}\n' notes-api/version
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth list

Expected:

pod/frontend2 created
pod/frontend2 condition met
NAME        SECURITY IDENTITY   ENDPOINT STATE   IPV4         IPV6
frontend    38567               ready            10.0.1.183
frontend2   38567               ready            10.0.1.79
SRC IDENTITY   DST IDENTITY   REMOTE NODE ID   AUTH TYPE   EXPIRATION
61704          51657          47089            spire       2026-11-19 01:44:45 +0000 UTC
61704          38567          0                spire       2026-11-19 01:44:44 +0000 UTC
0.5.1
 200
SRC IDENTITY   DST IDENTITY   REMOTE NODE ID   AUTH TYPE   EXPIRATION
61704          51657          47089            spire       2026-11-19 01:44:45 +0000 UTC
61704          38567          0                spire       2026-11-19 01:44:44 +0000 UTC

No new entry, no “Authentication required” drop: authentication happens per identity pair, not per Pod or connection (roadmap: “Per-connection handshake – TODO”). Whoever carries the labels of an authorized Pod inherits its clearance – another reason to protect labels and ServiceAccounts with RBAC and admission control.

Save your progress? Sign up for free

Cleanup

Mutual Authentication is beta and occupies a SPIRE agent on every node – back to the state before Day 14. The backup from Drill 5 contains no authentication values; without --reuse-values, the chart defaults apply again (both off), and Helm removes server, agents and RBAC:

k delete namespace tag14
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart deployment/cilium-operator
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
k delete namespace cilium-spire --ignore-not-found
cilium config view | grep -E '^mesh-auth-enabled'
k config set-context --current --namespace=default

Expected (abridged): GLEICH, mesh-auth-enabled false; the namespace cilium-spire including its PVC is gone.

Warning: Do not reset via --reuse-values --set authentication=null – in the lab test, the chart then keeps rendering with Mutual Authentication and SPIRE.

Exam Tip

  • Identity = labels + namespace + ServiceAccount. For “only Pods with ServiceAccount X”, match io.cilium.k8s.policy.serviceaccount: X in fromEndpoints.
  • L7 rules live under toPorts[].rules.http (method, path as regex, header); an L7 no is a 403, an L3/L4 no is a timeout – both visible in hubble observe.
  • authentication.mode: required sits at the level of fromEndpoints in the ingress rule; drop reason “Authentication required”. Docs: docs.cilium.io “Mutual Authentication (Beta)”.

Enterprise Note

  • Cilium Mutual Authentication is beta: no per-connection handshake, cannot be combined with ClusterMesh, tested only with SPIRE. For compliance-grade mTLS today, rather use a mesh (Istio Ambient, Linkerd) or Cilium’s ztunnel integration (also beta).
  • Always combine Mutual Authentication with WireGuard or IPsec – it does not replace encryption.
  • SPIRE in production: a dedicated HA server with an external database, an upstream CA from Vault or your PKI, monitoring of SVID issuance; an outage means new identity pairs no longer get through.
  • Labels are security attributes: use an admission policy (Kyverno, VAP) to prevent teams from using other teams’ app labels or ServiceAccounts.
  • The identity labels can be restricted with the Helm value labels – fewer identities, less policy load in large clusters.