Day 4: Cilium – L7, FQDN and Hubble
Goal: Build
cilium-labwith Cilium and Hubble and write policies that NetworkPolicy cannot express: HTTP methods and paths, domains, entities. · Duration: approx. 120 min · Prerequisite: Day 3
Briefing
- Cilium replaces kindnet and kube-proxy with eBPF. Every Pod gets a security identity from its labels, not from its IP.
CiliumNetworkPolicy(CNP) can do everything NetworkPolicy can, plus L7 rules (HTTP),toFQDNs, entities (world,host,kube-apiserver) and explicit denies.- L7 and DNS rules run through a proxy (Envoy or the DNS proxy in the agent). An L7 denial is not a timeout but HTTP 403.
- Hubble shows every decision:
FORWARDED,DROPPED, policy verdicts, HTTP flows.
client ──► frontend ──GET /api/*──► api ──► db Hubble Relay ◄── agent (per node)
✘ POST, /admin (403) │
client ──► example.com ✔ kubernetes.io ✘ hubble observe / Hubble UI
Drill 1: Cluster without a CNI
You created the configuration
~/guard-lab/kind/cilium-lab.yaml on Day 0:
disableDefaultCNI: true, kubeProxyMode: none,
one control plane node and two worker nodes.
mkdir -p ~/guard-lab/tag04 ~/guard-lab/code/cilium && cd ~/guard-lab/tag04
kind create cluster --config ~/guard-lab/kind/cilium-lab.yaml
kubectl get nodes
kubectl -n kube-system get dsExpected (abridged, may differ depending on version):
✓ Ensuring node image (kindest/node:v1.37.0)
Set kubectl context to "kind-cilium-lab"
NAME STATUS ROLES AGE VERSION
cilium-lab-control-plane NotReady control-plane 45s v1.37.0
cilium-lab-worker NotReady <none> 25s v1.37.0
cilium-lab-worker2 NotReady <none> 25s v1.37.0
No resources found in kube-system namespace.
NotReady is correct: Without a CNI, no Pod network can
come up. And there is no kube-proxy DaemonSet – Services
only work once Cilium takes them over.
Save your progress? Sign up for free
Drill 2: Cilium via Helm with Hubble
helm repo add cilium https://helm.cilium.io
helm repo update cilium
cat > ~/guard-lab/code/cilium/values-kind.yaml <<'EOF'
# Cilium 1.20 for kind (cilium-lab), Hubble with Relay and UI
kubeProxyReplacement: true
k8sServiceHost: cilium-lab-control-plane # API server directly, no kube-proxy
k8sServicePort: 6443
ipam:
mode: kubernetes # Pod CIDR from the node object (kind: 10.244.x.0/24)
operator:
replicas: 1 # lab: one operator is enough
hubble:
enabled: true
relay:
enabled: true
ui:
enabled: true
EOF
helm install cilium cilium/cilium --version 1.20.2 -n kube-system \
-f ~/guard-lab/code/cilium/values-kind.yamlWhy k8sServiceHost? Without kube-proxy there is no
Service IP for kubernetes.default yet at startup. The agent
has to find the API server directly. The container name
cilium-lab-control-plane is resolvable via DNS in the kind
Docker network.
You fetch the cilium and hubble CLIs as
binaries (macOS, Apple Silicon):
cd /tmp
curl -sLO https://github.com/cilium/cilium-cli/releases/download/v0.20.1/\
cilium-darwin-arm64.tar.gz
curl -sLO https://github.com/cilium/hubble/releases/download/v1.20.2/\
hubble-darwin-arm64.tar.gz
tar xzf cilium-darwin-arm64.tar.gz && tar xzf hubble-darwin-arm64.tar.gz
export PATH=/tmp:$PATH
cilium status --waitExpected (abridged, may differ depending on version):
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
DaemonSet cilium Desired: 3, Ready: 3/3
DaemonSet cilium-envoy Desired: 3, Ready: 3/3
Deployment cilium-operator Desired: 1, Ready: 1/1
Deployment hubble-relay Desired: 1, Ready: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1
kubectl get nodes
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg status | grep -E 'KubeProxyReplacement|Hubble|Proxy Status'Expected (abridged, may differ depending on version):
NAME STATUS ROLES AGE VERSION
cilium-lab-control-plane Ready control-plane 4m v1.37.0
cilium-lab-worker Ready <none> 4m v1.37.0
cilium-lab-worker2 Ready <none> 4m v1.37.0
KubeProxyReplacement: True [eth0 172.18.0.3 (Direct Routing)]
Proxy Status: OK, ip 10.244.0.95, 0 redirects active on ports 10000-20000
Hubble: Ok Current/Max Flows: 4095/4095 (100.00%)
Since Cilium 1.16, Envoy runs as a separate DaemonSet
cilium-envoy. In the agent, the debug tool is called
cilium-dbg, no longer cilium.
Now the demo app as usual:
kubectl apply -f ~/guard-lab/code/demo/
kubectl wait --for=condition=Ready pod --all -A --timeout=300s
alias cc=~/guard-lab/code/demo/conncheck.sh
cc default/client http://api.shopExpected (abridged, may differ depending on version):
✔ erlaubt default/client -> http://api.shop (HTTP 200)
Save your progress? Sign up for free
Drill 3: CiliumNetworkPolicy on L3/L4
The same rule as on Day 1, just in Cilium syntax:
shop/api accepts frontend on 8080 and the
scraper from monitoring on 9113.
cat > ~/guard-lab/code/cilium/api-l4.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: api-l4
namespace: shop
spec:
endpointSelector:
matchLabels:
app: api
ingress:
- fromEndpoints:
- matchLabels:
app: frontend # without a namespace label: only from shop
toPorts:
- ports:
- port: "8080"
protocol: TCP
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
app: scraper
toPorts:
- ports:
- port: "9113"
protocol: TCP
EOF
kubectl apply -f ~/guard-lab/code/cilium/api-l4.yaml
kubectl -n shop get cnp
cc shop/frontend http://api.shop
cc default/client http://api.shop
cc monitoring/scraper http://api.shop:9113/metrics
cc monitoring/scraper http://api.shopExpected (abridged, may differ depending on version):
ciliumnetworkpolicy.cilium.io/api-l4 created
NAME AGE VALID
api-l4 2s True
✔ erlaubt shop/frontend -> http://api.shop (HTTP 200)
✘ blockiert default/client -> http://api.shop (Timeout)
✔ erlaubt monitoring/scraper -> http://api.shop:9113/metrics (HTTP 200)
✘ blockiert monitoring/scraper -> http://api.shop (Timeout)
| NetworkPolicy | CiliumNetworkPolicy |
|---|---|
podSelector |
endpointSelector |
from[].podSelector |
fromEndpoints[].matchLabels |
namespaceSelector + podSelector |
label k8s:io.kubernetes.pod.namespace |
ports[].port: 8080 |
toPorts[].ports[].port: "8080" (string) |
Why “only from shop” without a namespace label? A namespaced CNP
automatically adds its own namespace to fromEndpoints. You
select namespace labels with the prefix
k8s:io.cilium.k8s.namespace.labels.
(e.g. …labels.purpose: monitoring).
Save your progress? Sign up for free
Drill 4: Hubble – who was dropped?
hubble observe -P sets up the port forward to the Relay
itself.
cc default/client http://api.shop >/dev/null
hubble observe -P --verdict DROPPED --namespace shop --last 5Expected (abridged, may differ depending on version):
Oct 10 08:12:03.512: default/client:51234 (ID:23412) <> shop/api-5ccd4587ff-gwhbn:8080
(ID:4012) Policy denied DROPPED (TCP Flags: SYN)
Oct 10 08:12:04.530: default/client:51234 (ID:23412) <> shop/api-5ccd4587ff-gwhbn:8080
(ID:4012) Policy denied DROPPED (TCP Flags: SYN)
Although the client calls api.shop:80, you see port
8080: Cilium translates the Service IP already at connect()
in the client Pod. Policies always apply to the Pod’s target port. Now
the policy decisions themselves, allowed and denied:
cc shop/frontend http://api.shop >/dev/null
hubble observe -P --type policy-verdict --to-namespace shop --last 4 \
--print-policy-namesExpected (abridged, may differ depending on version):
... default/client:51234 (ID:23412) <> shop/api-5ccd4587ff-gwhbn:8080 (ID:4012)
policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
... shop/frontend-5f9bb6b549-8hpqx:40112 (ID:9921) -> shop/api-5ccd4587ff-gwhbn:8080
(ID:4012) policy-verdict:L3-L4 INGRESS ALLOWED (TCP Flags: SYN)
policy-verdict:none means: no rule matched, default deny
took effect. L3-L4 names the level of the rule that allowed
it. --print-policy-names additionally appends the names of
the policies involved. Useful filters:
| Filter | Purpose |
|---|---|
--from-pod default/client |
only flows from this source Pod |
--to-port 8080 |
only one target port |
--protocol http |
only L7 HTTP flows (Drill 5) |
-f |
follow live |
Tip: The IDs are security identities.
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg identity get 4012shows the labels behind them.
Save your progress? Sign up for free
Drill 5: L7 – only GET on /api/*
The frontend should use the api read-only
under /api/…. Same policy name, so apply
replaces the L4 version.
cat > ~/guard-lab/code/cilium/api-l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: api-l4 # same name: replaces the L4 rule
namespace: shop
spec:
endpointSelector:
matchLabels:
app: api
ingress:
- fromEndpoints:
- matchLabels:
app: frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: GET
path: "/api/.*" # regex, anchored to the whole path
- method: GET
path: "/healthz"
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
app: scraper
toPorts:
- ports:
- port: "9113"
protocol: TCP
EOF
kubectl apply -f ~/guard-lab/code/cilium/api-l7.yaml
cc shop/frontend http://api.shop/api/orders
cc shop/frontend http://api.shop/admin
kubectl -n shop exec deploy/frontend -- \
curl -s -X POST http://api.shop/api/orders
cc default/client http://frontend.shop/api/ordersExpected (abridged, may differ depending on version):
ciliumnetworkpolicy.cilium.io/api-l4 configured
✔ erlaubt shop/frontend -> http://api.shop/api/orders (HTTP 200)
✔ erlaubt shop/frontend -> http://api.shop/admin (HTTP 403)
Access denied
✔ erlaubt default/client -> http://frontend.shop/api/orders (HTTP 200)
Watch out for line 2: conncheck.sh reports “erlaubt”
(allowed), because the TCP connection was established. The rejection
comes as HTTP 403 from the Envoy proxy. For L7 tests you check the
status code. The last call goes through the frontend, which
internally sends GET /api/orders to the api –
allowed.
hubble observe -P --protocol http --to-namespace shop --last 4Expected (abridged, may differ depending on version):
... shop/frontend-5f9bb6b549-8hpqx:40500 (ID:9921) -> shop/api-5ccd4587ff-gwhbn:8080
(ID:4012) http-request DROPPED (HTTP/1.1 POST http://api.shop/api/orders)
... shop/frontend-5f9bb6b549-8hpqx:40512 (ID:9921) -> shop/api-5ccd4587ff-gwhbn:8080
(ID:4012) http-request FORWARDED (HTTP/1.1 GET http://api.shop/api/orders)
... shop/api-5ccd4587ff-gwhbn:8080 (ID:4012) <- shop/frontend-5f9bb6b549-8hpqx:40512
(ID:9921) http-response FORWARDED (HTTP/1.1 200 1ms (GET http://api.shop/api/orders))
With L7 rules, Hubble shows you method, URL and status code – without a sidecar and without app logs.
Warning: Allow rules add up in Cilium too. If a second policy opens the same port on L4, the L7 rule has no effect: an L4 allow permits all paths.
Save your progress? Sign up for free
Drill 6: toFQDNs – egress by domain
The client may go outside, but only to
example.com and *.github.com on 443.
cat > ~/guard-lab/code/cilium/client-fqdn.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: client-fqdn
namespace: default
spec:
endpointSelector:
matchLabels:
app: client
egress:
- toEndpoints: # DNS via Cilium's DNS proxy
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
- toFQDNs:
- matchName: example.com
- matchPattern: "*.github.com"
toPorts:
- ports:
- port: "443"
protocol: TCP
EOF
kubectl apply -f ~/guard-lab/code/cilium/client-fqdn.yaml
TIMEOUT=5 cc default/client https://example.com
TIMEOUT=5 cc default/client https://api.github.com
TIMEOUT=5 cc default/client https://kubernetes.io
cc default/client http://frontend.shopExpected (abridged, may differ depending on version):
ciliumnetworkpolicy.cilium.io/client-fqdn created
✔ erlaubt default/client -> https://example.com (HTTP 200)
✔ erlaubt default/client -> https://api.github.com (HTTP 200)
✘ blockiert default/client -> https://kubernetes.io (Timeout)
✘ blockiert default/client -> http://frontend.shop (Timeout)
Why is the DNS rule with rules.dns needed? Cilium only
learns the IPs for a name when the DNS response passes through its DNS
proxy. Without rules.dns, toFQDNs stays empty.
The proxy fills a cache of name → IP with a TTL:
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg fqdn cache list | grep -E 'example|github'
hubble observe -P --from-pod default/client --verdict DROPPED --last 2Expected (abridged, may differ depending on version):
12 example.com. 23.192.228.80 2026-10-10T08:31:02Z ...
12 api.github.com. 140.82.121.6 2026-10-10T08:30:45Z ...
... default/client:39822 (ID:23412) -> kubernetes.io:443 (world) Policy denied
DROPPED (TCP Flags: SYN)
fqdn cache list only shows entries from the node on
which the ds/cilium Pod runs that kubectl exec
picked. If the client is elsewhere, use the agent on its
node.
Save your progress? Sign up for free
Drill 7: Entities instead of IP lists
On Day 2 you opened up the API server via ipBlock – with
a fixed IP. Cilium knows the API server as an entity, no matter which IP
it has.
cc default/client https://kubernetes.default
cat > ~/guard-lab/code/cilium/client-entities.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: client-entities
namespace: default
spec:
endpointSelector:
matchLabels:
app: client
egress:
- toEntities:
- kube-apiserver # no matter which IP the API server currently has
toPorts:
- ports:
- port: "6443"
protocol: TCP
- toEntities:
- host # the node itself (e.g. node-local agents)
EOF
kubectl apply -f ~/guard-lab/code/cilium/client-entities.yaml
cc default/client https://kubernetes.defaultExpected (abridged, may differ depending on version):
✘ blockiert default/client -> https://kubernetes.default (Timeout)
ciliumnetworkpolicy.cilium.io/client-entities created
✔ erlaubt default/client -> https://kubernetes.default (HTTP 403)
HTTP 403 comes from the API server itself (anonymous access); the connection is established. The port is 6443, not 443: Here too, the port after the Service translation is what counts. The most important entities:
| Entity | Meaning |
|---|---|
world |
everything outside the cluster (practically
0.0.0.0/0) |
cluster |
all endpoints in the cluster incl. nodes |
host / remote-node |
the node itself / all other nodes |
kube-apiserver |
the API server, even outside the cluster |
Security:
toEntities: [world]is convenient and almost always too broad. For egress to the internet, usetoFQDNs(Drill 6) ortoCIDRwith specific networks.
Save your progress? Sign up for free
Checkpoint
cc=~/guard-lab/code/demo/conncheck.sh
kubectl config current-context | grep -q kind-cilium-lab \
&& echo "✔ Kontext kind-cilium-lab" || echo "✘ falscher Kontext"
cilium status >/dev/null 2>&1 && echo "✔ Cilium OK" || echo "✘ Cilium nicht OK"
kubectl -n kube-system get ds kube-proxy >/dev/null 2>&1 \
&& echo "✘ kube-proxy läuft noch" || echo "✔ kein kube-proxy"
while read -r src url want; do
"$cc" "$src" "$url" "$want" >/dev/null \
&& echo "✔ $src -> $url ($want)" || echo "✘ $src -> $url (erwartet: $want)"
done <<'EOF'
shop/frontend http://api.shop/api/x allow
default/client http://api.shop deny
monitoring/scraper http://api.shop:9113/metrics allow
default/client https://example.com allow
default/client https://kubernetes.io deny
EOF
code=$(kubectl -n shop exec deploy/frontend -- curl -s -o /dev/null \
-w '%{http_code}' -X POST http://api.shop/api/x)
[ "$code" = 403 ] && echo "✔ POST -> 403" || echo "✘ POST -> $code"Challenge
The scraper should only be allowed
GET /metrics on port 9113. GET / on 9113 must
end with 403; the api on 8080 stays as in Drill 5.
Hint: Do not write a second policy for this – the L4
allow for 9113 in api-l4 would override your L7 rule.
Solution
cd ~/guard-lab/code/cilium
cp api-l7.yaml scraper-l7.yaml
cat >> scraper-l7.yaml <<'EOF'
rules:
http:
- method: GET
path: "/metrics"
EOF
kubectl apply -f scraper-l7.yaml
cc monitoring/scraper http://api.shop:9113/metrics
cc monitoring/scraper http://api.shop:9113/Expected (abridged, may differ depending on version):
ciliumnetworkpolicy.cilium.io/api-l4 configured
✔ erlaubt monitoring/scraper -> http://api.shop:9113/metrics (HTTP 200)
✔ erlaubt monitoring/scraper -> http://api.shop:9113/ (HTTP 403)
The cat >> appends the rules to the
last toPorts entry – the one for 9113. Check the
indentation with tail -12 scraper-l7.yaml before you
apply.
Save your progress? Sign up for free
Enterprise Note
- The Cilium version, chart values and Envoy mode belong in Git;
upgrades only minor by minor, with
cilium statusandcilium connectivity testin between. toFQDNsdepends on the DNS proxy. If the agent fails or the proxy is overloaded, DNS fails for all Pods with FQDN rules – monitor the proxy metrics.- L7 rules cost latency and CPU in Envoy. Only use them where the method or path really matters (admin endpoints, write access).
- Hubble flows are sensitive data (URLs, IPs). Run Relay and UI only with TLS and access control, export to a SIEM with a retention period.
- Clean up:
kind delete cluster --name cilium-lab– or keep it running for Days 5 and 6.