Skip to content
Hands-on IT IT Drills

Day 4: Cilium – L7, FQDN and Hubble

Goal: Build cilium-lab with Cilium and Hubble and write policies that NetworkPolicy cannot express: HTTP methods and paths, domains, entities. · Duration: approx. 120 min · Prerequisite: Day 3

Briefing

  • Cilium replaces kindnet and kube-proxy with eBPF. Every Pod gets a security identity from its labels, not from its IP.
  • CiliumNetworkPolicy (CNP) can do everything NetworkPolicy can, plus L7 rules (HTTP), toFQDNs, entities (world, host, kube-apiserver) and explicit denies.
  • L7 and DNS rules run through a proxy (Envoy or the DNS proxy in the agent). An L7 denial is not a timeout but HTTP 403.
  • Hubble shows every decision: FORWARDED, DROPPED, policy verdicts, HTTP flows.
 client ──► frontend ──GET /api/*──► api ──► db        Hubble Relay ◄── agent (per node)
                  ✘ POST, /admin (403)                      │
 client ──► example.com ✔   kubernetes.io ✘          hubble observe / Hubble UI

Drill 1: Cluster without a CNI

You created the configuration ~/guard-lab/kind/cilium-lab.yaml on Day 0: disableDefaultCNI: true, kubeProxyMode: none, one control plane node and two worker nodes.

mkdir -p ~/guard-lab/tag04 ~/guard-lab/code/cilium && cd ~/guard-lab/tag04
kind create cluster --config ~/guard-lab/kind/cilium-lab.yaml
kubectl get nodes
kubectl -n kube-system get ds

Expected (abridged, may differ depending on version):

 ✓ Ensuring node image (kindest/node:v1.37.0)
Set kubectl context to "kind-cilium-lab"
NAME                       STATUS     ROLES           AGE   VERSION
cilium-lab-control-plane   NotReady   control-plane   45s   v1.37.0
cilium-lab-worker          NotReady   <none>          25s   v1.37.0
cilium-lab-worker2         NotReady   <none>          25s   v1.37.0
No resources found in kube-system namespace.

NotReady is correct: Without a CNI, no Pod network can come up. And there is no kube-proxy DaemonSet – Services only work once Cilium takes them over.

Save your progress? Sign up for free

Drill 2: Cilium via Helm with Hubble

helm repo add cilium https://helm.cilium.io
helm repo update cilium
cat > ~/guard-lab/code/cilium/values-kind.yaml <<'EOF'
# Cilium 1.20 for kind (cilium-lab), Hubble with Relay and UI
kubeProxyReplacement: true
k8sServiceHost: cilium-lab-control-plane   # API server directly, no kube-proxy
k8sServicePort: 6443
ipam:
  mode: kubernetes          # Pod CIDR from the node object (kind: 10.244.x.0/24)
operator:
  replicas: 1               # lab: one operator is enough
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
EOF
helm install cilium cilium/cilium --version 1.20.2 -n kube-system \
  -f ~/guard-lab/code/cilium/values-kind.yaml

Why k8sServiceHost? Without kube-proxy there is no Service IP for kubernetes.default yet at startup. The agent has to find the API server directly. The container name cilium-lab-control-plane is resolvable via DNS in the kind Docker network.

You fetch the cilium and hubble CLIs as binaries (macOS, Apple Silicon):

cd /tmp
curl -sLO https://github.com/cilium/cilium-cli/releases/download/v0.20.1/\
cilium-darwin-arm64.tar.gz
curl -sLO https://github.com/cilium/hubble/releases/download/v1.20.2/\
hubble-darwin-arm64.tar.gz
tar xzf cilium-darwin-arm64.tar.gz && tar xzf hubble-darwin-arm64.tar.gz
export PATH=/tmp:$PATH
cilium status --wait

Expected (abridged, may differ depending on version):

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 3, Ready: 3/3
DaemonSet              cilium-envoy             Desired: 3, Ready: 3/3
Deployment             cilium-operator          Desired: 1, Ready: 1/1
Deployment             hubble-relay             Desired: 1, Ready: 1/1
Deployment             hubble-ui                Desired: 1, Ready: 1/1
kubectl get nodes
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg status | grep -E 'KubeProxyReplacement|Hubble|Proxy Status'

Expected (abridged, may differ depending on version):

NAME                       STATUS   ROLES           AGE   VERSION
cilium-lab-control-plane   Ready    control-plane   4m    v1.37.0
cilium-lab-worker          Ready    <none>          4m    v1.37.0
cilium-lab-worker2         Ready    <none>          4m    v1.37.0
KubeProxyReplacement:    True   [eth0   172.18.0.3 (Direct Routing)]
Proxy Status:            OK, ip 10.244.0.95, 0 redirects active on ports 10000-20000
Hubble:                  Ok   Current/Max Flows: 4095/4095 (100.00%)

Since Cilium 1.16, Envoy runs as a separate DaemonSet cilium-envoy. In the agent, the debug tool is called cilium-dbg, no longer cilium.

Now the demo app as usual:

kubectl apply -f ~/guard-lab/code/demo/
kubectl wait --for=condition=Ready pod --all -A --timeout=300s
alias cc=~/guard-lab/code/demo/conncheck.sh
cc default/client http://api.shop

Expected (abridged, may differ depending on version):

✔ erlaubt   default/client -> http://api.shop (HTTP 200)

Save your progress? Sign up for free

Drill 3: CiliumNetworkPolicy on L3/L4

The same rule as on Day 1, just in Cilium syntax: shop/api accepts frontend on 8080 and the scraper from monitoring on 9113.

cat > ~/guard-lab/code/cilium/api-l4.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-l4
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend          # without a namespace label: only from shop
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
  - fromEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: monitoring
        app: scraper
    toPorts:
    - ports:
      - port: "9113"
        protocol: TCP
EOF
kubectl apply -f ~/guard-lab/code/cilium/api-l4.yaml
kubectl -n shop get cnp
cc shop/frontend http://api.shop
cc default/client http://api.shop
cc monitoring/scraper http://api.shop:9113/metrics
cc monitoring/scraper http://api.shop

Expected (abridged, may differ depending on version):

ciliumnetworkpolicy.cilium.io/api-l4 created
NAME     AGE   VALID
api-l4   2s    True
✔ erlaubt   shop/frontend -> http://api.shop (HTTP 200)
✘ blockiert default/client -> http://api.shop (Timeout)
✔ erlaubt   monitoring/scraper -> http://api.shop:9113/metrics (HTTP 200)
✘ blockiert monitoring/scraper -> http://api.shop (Timeout)
NetworkPolicy CiliumNetworkPolicy
podSelector endpointSelector
from[].podSelector fromEndpoints[].matchLabels
namespaceSelector + podSelector label k8s:io.kubernetes.pod.namespace
ports[].port: 8080 toPorts[].ports[].port: "8080" (string)

Why “only from shop” without a namespace label? A namespaced CNP automatically adds its own namespace to fromEndpoints. You select namespace labels with the prefix k8s:io.cilium.k8s.namespace.labels. (e.g. …labels.purpose: monitoring).

Save your progress? Sign up for free

Drill 4: Hubble – who was dropped?

hubble observe -P sets up the port forward to the Relay itself.

cc default/client http://api.shop >/dev/null
hubble observe -P --verdict DROPPED --namespace shop --last 5

Expected (abridged, may differ depending on version):

Oct 10 08:12:03.512: default/client:51234 (ID:23412) <> shop/api-5ccd4587ff-gwhbn:8080
  (ID:4012) Policy denied DROPPED (TCP Flags: SYN)
Oct 10 08:12:04.530: default/client:51234 (ID:23412) <> shop/api-5ccd4587ff-gwhbn:8080
  (ID:4012) Policy denied DROPPED (TCP Flags: SYN)

Although the client calls api.shop:80, you see port 8080: Cilium translates the Service IP already at connect() in the client Pod. Policies always apply to the Pod’s target port. Now the policy decisions themselves, allowed and denied:

cc shop/frontend http://api.shop >/dev/null
hubble observe -P --type policy-verdict --to-namespace shop --last 4 \
  --print-policy-names

Expected (abridged, may differ depending on version):

... default/client:51234 (ID:23412) <> shop/api-5ccd4587ff-gwhbn:8080 (ID:4012)
  policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
... shop/frontend-5f9bb6b549-8hpqx:40112 (ID:9921) -> shop/api-5ccd4587ff-gwhbn:8080
  (ID:4012) policy-verdict:L3-L4 INGRESS ALLOWED (TCP Flags: SYN)

policy-verdict:none means: no rule matched, default deny took effect. L3-L4 names the level of the rule that allowed it. --print-policy-names additionally appends the names of the policies involved. Useful filters:

Filter Purpose
--from-pod default/client only flows from this source Pod
--to-port 8080 only one target port
--protocol http only L7 HTTP flows (Drill 5)
-f follow live

Tip: The IDs are security identities. kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg identity get 4012 shows the labels behind them.

Save your progress? Sign up for free

Drill 5: L7 – only GET on /api/*

The frontend should use the api read-only under /api/…. Same policy name, so apply replaces the L4 version.

cat > ~/guard-lab/code/cilium/api-l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-l4                 # same name: replaces the L4 rule
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/api/.*"      # regex, anchored to the whole path
        - method: GET
          path: "/healthz"
  - fromEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: monitoring
        app: scraper
    toPorts:
    - ports:
      - port: "9113"
        protocol: TCP
EOF
kubectl apply -f ~/guard-lab/code/cilium/api-l7.yaml
cc shop/frontend http://api.shop/api/orders
cc shop/frontend http://api.shop/admin
kubectl -n shop exec deploy/frontend -- \
  curl -s -X POST http://api.shop/api/orders
cc default/client http://frontend.shop/api/orders

Expected (abridged, may differ depending on version):

ciliumnetworkpolicy.cilium.io/api-l4 configured
✔ erlaubt   shop/frontend -> http://api.shop/api/orders (HTTP 200)
✔ erlaubt   shop/frontend -> http://api.shop/admin (HTTP 403)
Access denied
✔ erlaubt   default/client -> http://frontend.shop/api/orders (HTTP 200)

Watch out for line 2: conncheck.sh reports “erlaubt” (allowed), because the TCP connection was established. The rejection comes as HTTP 403 from the Envoy proxy. For L7 tests you check the status code. The last call goes through the frontend, which internally sends GET /api/orders to the api – allowed.

hubble observe -P --protocol http --to-namespace shop --last 4

Expected (abridged, may differ depending on version):

... shop/frontend-5f9bb6b549-8hpqx:40500 (ID:9921) -> shop/api-5ccd4587ff-gwhbn:8080
  (ID:4012) http-request DROPPED (HTTP/1.1 POST http://api.shop/api/orders)
... shop/frontend-5f9bb6b549-8hpqx:40512 (ID:9921) -> shop/api-5ccd4587ff-gwhbn:8080
  (ID:4012) http-request FORWARDED (HTTP/1.1 GET http://api.shop/api/orders)
... shop/api-5ccd4587ff-gwhbn:8080 (ID:4012) <- shop/frontend-5f9bb6b549-8hpqx:40512
  (ID:9921) http-response FORWARDED (HTTP/1.1 200 1ms (GET http://api.shop/api/orders))

With L7 rules, Hubble shows you method, URL and status code – without a sidecar and without app logs.

Warning: Allow rules add up in Cilium too. If a second policy opens the same port on L4, the L7 rule has no effect: an L4 allow permits all paths.

Save your progress? Sign up for free

Drill 6: toFQDNs – egress by domain

The client may go outside, but only to example.com and *.github.com on 443.

cat > ~/guard-lab/code/cilium/client-fqdn.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: client-fqdn
  namespace: default
spec:
  endpointSelector:
    matchLabels:
      app: client
  egress:
  - toEndpoints:               # DNS via Cilium's DNS proxy
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toFQDNs:
    - matchName: example.com
    - matchPattern: "*.github.com"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
EOF
kubectl apply -f ~/guard-lab/code/cilium/client-fqdn.yaml
TIMEOUT=5 cc default/client https://example.com
TIMEOUT=5 cc default/client https://api.github.com
TIMEOUT=5 cc default/client https://kubernetes.io
cc default/client http://frontend.shop

Expected (abridged, may differ depending on version):

ciliumnetworkpolicy.cilium.io/client-fqdn created
✔ erlaubt   default/client -> https://example.com (HTTP 200)
✔ erlaubt   default/client -> https://api.github.com (HTTP 200)
✘ blockiert default/client -> https://kubernetes.io (Timeout)
✘ blockiert default/client -> http://frontend.shop (Timeout)

Why is the DNS rule with rules.dns needed? Cilium only learns the IPs for a name when the DNS response passes through its DNS proxy. Without rules.dns, toFQDNs stays empty. The proxy fills a cache of name → IP with a TTL:

kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg fqdn cache list | grep -E 'example|github'
hubble observe -P --from-pod default/client --verdict DROPPED --last 2

Expected (abridged, may differ depending on version):

12    example.com.      23.192.228.80    2026-10-10T08:31:02Z   ...
12    api.github.com.   140.82.121.6     2026-10-10T08:30:45Z   ...
... default/client:39822 (ID:23412) -> kubernetes.io:443 (world) Policy denied
  DROPPED (TCP Flags: SYN)

fqdn cache list only shows entries from the node on which the ds/cilium Pod runs that kubectl exec picked. If the client is elsewhere, use the agent on its node.

Save your progress? Sign up for free

Drill 7: Entities instead of IP lists

On Day 2 you opened up the API server via ipBlock – with a fixed IP. Cilium knows the API server as an entity, no matter which IP it has.

cc default/client https://kubernetes.default
cat > ~/guard-lab/code/cilium/client-entities.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: client-entities
  namespace: default
spec:
  endpointSelector:
    matchLabels:
      app: client
  egress:
  - toEntities:
    - kube-apiserver           # no matter which IP the API server currently has
    toPorts:
    - ports:
      - port: "6443"
        protocol: TCP
  - toEntities:
    - host                     # the node itself (e.g. node-local agents)
EOF
kubectl apply -f ~/guard-lab/code/cilium/client-entities.yaml
cc default/client https://kubernetes.default

Expected (abridged, may differ depending on version):

✘ blockiert default/client -> https://kubernetes.default (Timeout)
ciliumnetworkpolicy.cilium.io/client-entities created
✔ erlaubt   default/client -> https://kubernetes.default (HTTP 403)

HTTP 403 comes from the API server itself (anonymous access); the connection is established. The port is 6443, not 443: Here too, the port after the Service translation is what counts. The most important entities:

Entity Meaning
world everything outside the cluster (practically 0.0.0.0/0)
cluster all endpoints in the cluster incl. nodes
host / remote-node the node itself / all other nodes
kube-apiserver the API server, even outside the cluster

Security: toEntities: [world] is convenient and almost always too broad. For egress to the internet, use toFQDNs (Drill 6) or toCIDR with specific networks.

Save your progress? Sign up for free

Checkpoint

cc=~/guard-lab/code/demo/conncheck.sh
kubectl config current-context | grep -q kind-cilium-lab \
  && echo "✔ Kontext kind-cilium-lab" || echo "✘ falscher Kontext"
cilium status >/dev/null 2>&1 && echo "✔ Cilium OK" || echo "✘ Cilium nicht OK"
kubectl -n kube-system get ds kube-proxy >/dev/null 2>&1 \
  && echo "✘ kube-proxy läuft noch" || echo "✔ kein kube-proxy"
while read -r src url want; do
  "$cc" "$src" "$url" "$want" >/dev/null \
    && echo "✔ $src -> $url ($want)" || echo "✘ $src -> $url (erwartet: $want)"
done <<'EOF'
shop/frontend http://api.shop/api/x allow
default/client http://api.shop deny
monitoring/scraper http://api.shop:9113/metrics allow
default/client https://example.com allow
default/client https://kubernetes.io deny
EOF
code=$(kubectl -n shop exec deploy/frontend -- curl -s -o /dev/null \
  -w '%{http_code}' -X POST http://api.shop/api/x)
[ "$code" = 403 ] && echo "✔ POST -> 403" || echo "✘ POST -> $code"

Challenge

The scraper should only be allowed GET /metrics on port 9113. GET / on 9113 must end with 403; the api on 8080 stays as in Drill 5.

Hint: Do not write a second policy for this – the L4 allow for 9113 in api-l4 would override your L7 rule.

Solution

cd ~/guard-lab/code/cilium
cp api-l7.yaml scraper-l7.yaml
cat >> scraper-l7.yaml <<'EOF'
      rules:
        http:
        - method: GET
          path: "/metrics"
EOF
kubectl apply -f scraper-l7.yaml
cc monitoring/scraper http://api.shop:9113/metrics
cc monitoring/scraper http://api.shop:9113/

Expected (abridged, may differ depending on version):

ciliumnetworkpolicy.cilium.io/api-l4 configured
✔ erlaubt   monitoring/scraper -> http://api.shop:9113/metrics (HTTP 200)
✔ erlaubt   monitoring/scraper -> http://api.shop:9113/ (HTTP 403)

The cat >> appends the rules to the last toPorts entry – the one for 9113. Check the indentation with tail -12 scraper-l7.yaml before you apply.

Save your progress? Sign up for free

Enterprise Note

  • The Cilium version, chart values and Envoy mode belong in Git; upgrades only minor by minor, with cilium status and cilium connectivity test in between.
  • toFQDNs depends on the DNS proxy. If the agent fails or the proxy is overloaded, DNS fails for all Pods with FQDN rules – monitor the proxy metrics.
  • L7 rules cost latency and CPU in Envoy. Only use them where the method or path really matters (admin endpoints, write access).
  • Hubble flows are sensitive data (URLs, IPs). Run Relay and UI only with TLS and access control, export to a SIEM with a retention period.
  • Clean up: kind delete cluster --name cilium-lab – or keep it running for Days 5 and 6.