Skip to content
Hands-on IT IT Drills

Day 2: NetworkPolicies and CiliumNetworkPolicy

Goal: You block Pod traffic by default, open only the paths you need, keep Pods away from node endpoints and cloud metadata, and use Cilium for L7 and DNS rules. · Duration: approx. 90 min · Prerequisite: Day 1

Briefing

  • CKS: Cluster Setup (15 %) – “Use network security policies to restrict cluster level access” and “Protect node metadata and endpoints”.
  • Without a policy, every Pod talks to every other Pod – and to the kubelet, the API server and the metadata IP 169.254.169.254, through which cloud VMs hand out credentials.
  • Policies are additive. The standard NetworkPolicy has no “deny”; Cilium does.
  • Cilium treats nodes as an identity of their own: ipBlock does not cover them (Drill 4).
 tag02-other/outsider ──┐
 tag02/client ──────────┴─> tag02/web :80
        └──X──> 169.254.169.254 · Nodes :10250/:6443 · Internet (targeted only)

Drill 1: The test field

mkdir -p ~/cks/tag02 && cd ~/cks/tag02
k create ns tag02
k create ns tag02-other
k config set-context --current --namespace=tag02
k create deployment web --image=nginx:1.29 --replicas=2 --port=80
k expose deployment web --port=80
k run client --image=busybox:1.37 -- sleep 3600
k run outsider -n tag02-other --image=busybox:1.37 -- sleep 3600
k rollout status deployment web
k exec client -- wget -qO- -T2 http://web | grep title
k exec -n tag02-other outsider -- wget -qO- -T2 http://web.tag02 | grep title
k exec client -- nc -zv -w2 172.16.0.228 10250
k exec client -- nc -zv -w2 172.16.0.126 6443
k exec client -- nc -zv -w2 169.254.169.254 80

Expected (shortened):

deployment "web" successfully rolled out
<title>Welcome to nginx!</title>
<title>Welcome to nginx!</title>
172.16.0.228 (172.16.0.228:10250) open
172.16.0.126 (172.16.0.126:6443) open
nc: 169.254.169.254 (169.254.169.254:80): Network is unreachable
command terminated with exit code 1

Every Pod reaches the kubelet and API server of the nodes. There is no metadata service in the lab, hence Network is unreachable – in AWS, Azure or GCP you would get an answer at this point, often with credentials of the node role. Always set timeouts (-T2, -w2), otherwise you wait forever when traffic is blocked.

Save your progress? Sign up for free

Drill 2: Default deny in both directions

cat > default-deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
EOF
k apply -f default-deny.yaml
k exec client -- wget -qO- -T2 http://web
k exec -n tag02-other outsider -- wget -qO- -T2 http://web.tag02
k exec client -- nc -zv -w2 172.16.0.228 10250
k exec client -- nc -zv -w2 169.254.169.254 80

Expected:

networkpolicy.networking.k8s.io/default-deny created
wget: bad address 'web'
command terminated with exit code 1
wget: download timed out
command terminated with exit code 1
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
nc: 169.254.169.254 (169.254.169.254:80): Connection timed out
command terminated with exit code 1

podSelector: {} selects all Pods in the namespace; without ingress/egress lists, nothing is allowed. bad address shows: DNS is egress too. Network is unreachable turns into timed out – Cilium already drops the packets at the Pod interface, before they leave the node.

Save your progress? Sign up for free

Drill 3: Allow DNS and targeted paths

cat > allow-dns.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
EOF
k apply -f allow-dns.yaml
k exec client -- nslookup web.tag02.svc.cluster.local
k exec client -- wget -qO- -T2 http://web

Expected:

networkpolicy.networking.k8s.io/allow-dns created
Server:    10.96.0.10
Address:   10.96.0.10:53

Name:      web.tag02.svc.cluster.local
Address:   10.99.244.41

wget: download timed out
command terminated with exit code 1

The name resolves, the connection stays closed. Inside a namespace with default deny, every path needs two permissions: egress at the sender, ingress at the receiver. On top, the namespace tag02-other is allowed in:

cat > web-policies.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-ingress
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          run: client
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: tag02-other
    ports:
    - port: 80
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: client-egress
spec:
  podSelector:
    matchLabels:
      run: client
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: web
    ports:
    - port: 80
EOF
k apply -f web-policies.yaml
k exec client -- wget -qO- -T2 http://web | grep title
k exec -n tag02-other outsider -- wget -qO- -T2 http://web.tag02 | grep title
k describe netpol web-ingress | sed -n '/Allowing ingress/,/Not affecting/p'

Expected:

networkpolicy.networking.k8s.io/web-ingress created
networkpolicy.networking.k8s.io/client-egress created
<title>Welcome to nginx!</title>
<title>Welcome to nginx!</title>
  Allowing ingress traffic:
    To Port: 80/TCP
    From:
      PodSelector: run=client
    From:
      NamespaceSelector: kubernetes.io/metadata.name=tag02-other
  Not affecting egress traffic

Two From: blocks mean OR: Pods with run=client from tag02 or any Pod from tag02-other. k run sets run: client automatically.

Save your progress? Sign up for free

Drill 4: Internet yes, metadata no – ipBlock with except

client should reach the internet, but never the metadata IP:

cat > client-world.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: client-world
spec:
  podSelector:
    matchLabels:
      run: client
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 169.254.169.254/32
EOF
k apply -f client-world.yaml
k exec client -- nc -zv -w3 kubernetes.io 443
k exec client -- nc -zv -w2 169.254.169.254 80
k exec client -- nc -zv -w2 172.16.0.228 10250
k exec client -- nc -zv -w2 172.16.0.126 6443

Expected:

networkpolicy.networking.k8s.io/client-world created
kubernetes.io (3.33.186.135:443) open
nc: 169.254.169.254 (169.254.169.254:80): Connection timed out
command terminated with exit code 1
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
nc: 172.16.0.126 (172.16.0.126:6443): Connection timed out
command terminated with exit code 1

Surprise: the nodes are blocked, even though 0.0.0.0/0 includes them. Cross-check with exactly the node network:

k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tmp-nodes
spec:
  podSelector:
    matchLabels:
      run: client
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 172.16.0.0/24
EOF
k exec client -- nc -zv -w2 172.16.0.228 10250
k delete netpol tmp-nodes

Expected:

networkpolicy.networking.k8s.io/tmp-nodes created
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
networkpolicy.networking.k8s.io "tmp-nodes" deleted from tag02 namespace

Cilium maps node IPs to the identities host and remote-node, and by default ipBlock only covers “world”. Other CNIs (Calico, for example) let 0.0.0.0/0 reach the nodes as well. That is why, in the exam: write the node network and the metadata IP explicitly into except – that works with every CNI.

Save your progress? Sign up for free

Drill 5: CiliumNetworkPolicy – L3/L4 and entities

Cilium policies in a namespace of their own, so that nothing mixes with Drills 2 to 4:

k create ns tag02-cilium
k -n tag02-cilium create deployment web --image=nginx:1.29 --port=80
k -n tag02-cilium expose deployment web --port=80
k -n tag02-cilium run client --image=busybox:1.37 -- sleep 3600
k -n tag02-cilium rollout status deployment web
cat > cnp-client.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: client-egress
  namespace: tag02-cilium
spec:
  endpointSelector:
    matchLabels:
      run: client
  egress:
  - toEndpoints:
    - matchLabels:
        app: web
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
  - toEntities:
    - kube-apiserver
    toPorts:
    - ports:
      - port: "6443"
        protocol: TCP
EOF
k apply -f cnp-client.yaml
k get cnp -n tag02-cilium

Expected:

ciliumnetworkpolicy.cilium.io/client-egress created
NAME            AGE   VALID
client-egress   3s    True

Always check VALID True: Cilium accepts faulty policies but does not enforce them. Ports are quoted. Now test:

k -n tag02-cilium exec client -- wget -qO- -T2 http://web | grep title
k -n tag02-cilium exec client -- nc -zv -w2 kubernetes.default 443
k -n tag02-cilium exec client -- nc -zv -w2 172.16.0.228 10250
k -n tag02-cilium exec client -- nc -zv -w2 kubernetes.io 443

Expected:

<title>Welcome to nginx!</title>
kubernetes.default (10.96.0.1:443) open
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
nc: kubernetes.io (3.33.186.135:443): Connection timed out
command terminated with exit code 1

A CNP egress section with rules switches the Pod to default deny. toEntities: kube-apiserver allows exactly the API server – no matter which node it runs on; the kubelet on worker1 stays closed. More entities: world, cluster, host, remote-node, all (k explain cnp.spec.egress.toEntities).

Save your progress? Sign up for free

Drill 6: L7 rule for HTTP

Standard NetworkPolicies only know ports. Cilium also checks method and path:

k -n tag02-cilium exec deploy/web -- \
  sh -c 'echo geheim > /usr/share/nginx/html/admin.html'
k -n tag02-cilium exec client -- wget -qO- -T2 http://web/admin.html
cat > cnp-web-l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: web-l7
  namespace: tag02-cilium
spec:
  endpointSelector:
    matchLabels:
      app: web
  ingress:
  - fromEndpoints:
    - matchLabels:
        run: client
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/"
        - method: GET
          path: "/index.html"
EOF
k apply -f cnp-web-l7.yaml
k -n tag02-cilium exec client -- wget -qO- -T2 http://web | grep title
k -n tag02-cilium exec client -- wget -qO- -T2 http://web/admin.html
k -n tag02-cilium exec client -- wget -qO- -T2 --post-data=x http://web/

Expected:

geheim
ciliumnetworkpolicy.cilium.io/web-l7 created
<title>Welcome to nginx!</title>
wget: server returned error: HTTP/1.1 403 Forbidden
command terminated with exit code 1
wget: server returned error: HTTP/1.1 403 Forbidden
command terminated with exit code 1

The 403 does not come from nginx but from Cilium’s Envoy proxy (DaemonSet cilium-envoy). path is a regular expression over the whole path: / does not allow /admin.html.

Save your progress? Sign up for free

Drill 7: toFQDNs – egress by name

External destinations change their IPs. Cilium learns them from the DNS responses – for that, the DNS rule must go through the Cilium DNS proxy (rules: dns):

cat > cnp-fqdn.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: client-fqdn
  namespace: tag02-cilium
spec:
  endpointSelector:
    matchLabels:
      run: client
  egress:
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toFQDNs:
    - matchName: kubernetes.io
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
EOF
k apply -f cnp-fqdn.yaml
k -n tag02-cilium exec client -- nc -zv -w3 kubernetes.io 443
k -n tag02-cilium exec client -- nc -zv -w3 kubernetes.io 80
k -n tag02-cilium exec client -- nc -zv -w3 github.com 443

Expected:

ciliumnetworkpolicy.cilium.io/client-fqdn created
kubernetes.io (3.33.186.135:443) open
nc: kubernetes.io (3.33.186.135:80): Connection timed out
command terminated with exit code 1
nc: github.com (140.82.121.3:443): Connection timed out
command terminated with exit code 1

Both CNPs select client and add up: web, DNS, the API server and kubernetes.io:443 are open, nothing else. Without the dns rule, toFQDNs would have no effect, because Cilium never sees the name resolution.

Save your progress? Sign up for free

Drill 8: Block metadata cluster-wide – deny rule

A real “deny” for all Pods, without sending them into default deny. For practice only for the namespace tag02-cilium (in production endpointSelector: {}):

k -n tag02-cilium run probe --image=busybox:1.37 -- sleep 3600
k -n tag02-cilium exec probe -- nc -zv -w2 169.254.169.254 80
cat > ccnp-metadata.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: tag02-deny-metadata
spec:
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: tag02-cilium
  enableDefaultDeny:
    egress: false
    ingress: false
  egressDeny:
  - toCIDR:
    - 169.254.169.254/32
EOF
k apply -f ccnp-metadata.yaml
k get ccnp
k -n tag02-cilium exec probe -- nc -zv -w2 169.254.169.254 80
k -n tag02-cilium exec probe -- nc -zv -w3 kubernetes.io 443

Expected:

nc: 169.254.169.254 (169.254.169.254:80): Network is unreachable
command terminated with exit code 1
ciliumclusterwidenetworkpolicy.cilium.io/tag02-deny-metadata created
NAME                  VALID
tag02-deny-metadata   True
nc: 169.254.169.254 (169.254.169.254:80): Connection timed out
command terminated with exit code 1
kubernetes.io (15.197.167.90:443) open

probe still reaches the internet; only the metadata IP is gone. enableDefaultDeny: false prevents the policy from blocking everything else. Deny rules take precedence over every allow – not even a later allow policy opens the metadata.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Block all incoming traffic in tag02-other. Target time: 45 s
  2. Show all Cilium policies in the cluster, namespaced and cluster-wide, with validity. Target time: 15 s
  3. Extend web-l7: client may additionally read all paths under /admin via GET. Target time: 60 s
  4. Which policy types and which exceptions does client-world have? Target time: 30 s

Solution

k apply -n tag02-other -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress
EOF
k get cnp,ccnp -A
vim cnp-web-l7.yaml

Add below the entry for /index.html:

        - method: GET
          path: "/admin.*"
k apply -f cnp-web-l7.yaml
k -n tag02-cilium exec client -- wget -qO- -T2 http://web/admin.html
k get netpol client-world \
  -o jsonpath='{.spec.policyTypes} {.spec.egress[0].to[0].ipBlock.except}{"\n"}'

Expected (shortened):

networkpolicy.networking.k8s.io/default-deny-ingress created
NAMESPACE      NAME                                          AGE   VALID
tag02-cilium   ciliumnetworkpolicy.cilium.io/client-egress   98s   True
tag02-cilium   ciliumnetworkpolicy.cilium.io/client-fqdn     68s   True
tag02-cilium   ciliumnetworkpolicy.cilium.io/web-l7          81s   True

NAMESPACE   NAME                                                           VALID
            ciliumclusterwidenetworkpolicy.cilium.io/tag02-deny-metadata   True
ciliumnetworkpolicy.cilium.io/web-l7 configured
geheim
["Egress"] ["169.254.169.254/32"]

Save your progress? Sign up for free

Checkpoint

k get netpol -n tag02 -o custom-columns=NAME:.metadata.name,TYPES:.spec.policyTypes

Expected:

NAME            TYPES
allow-dns       [Egress]
client-egress   [Egress]
client-world    [Egress]
default-deny    [Ingress Egress]
web-ingress     [Ingress]
k exec -n tag02 client -- nc -zv -w2 169.254.169.254 80

Expected: Connection timed out

k -n tag02-cilium exec client -- wget -qO- -T2 http://web/geheim.html

Expected: wget: server returned error: HTTP/1.1 403 Forbidden – the L7 rule takes effect.

k get ccnp tag02-deny-metadata

Expected: VALID = True

Challenge

client in tag02 should additionally reach the API server via kubernetes.default:443, but still no kubelet. A NetworkPolicy with ipBlock: 172.16.0.126/32 and port 6443 does not help here – find the way that works with Cilium.

Hint: Drill 4 explains why ipBlock fails; Drill 5 shows the entity.

Solution

k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: client-api
spec:
  endpointSelector:
    matchLabels:
      run: client
  egress:
  - toEntities:
    - kube-apiserver
    toPorts:
    - ports:
      - port: "6443"
        protocol: TCP
EOF
k exec client -- nc -zv -w2 kubernetes.default 443
k exec client -- nc -zv -w2 172.16.0.228 10250

Expected:

ciliumnetworkpolicy.cilium.io/client-api created
kubernetes.default (10.96.0.1:443) open
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1

The port is 6443, not 443: policies take effect after the Service translation. Clean up:

k config set-context --current --namespace=default
k delete ns tag02 tag02-other tag02-cilium
k delete ccnp tag02-deny-metadata

Save your progress? Sign up for free

Exam Tip

  • First find out which CNI is running (k get pods -n kube-system). Use Cilium objects only when the task asks for them; otherwise the standard NetworkPolicy.
  • Always pair egress blocks with a DNS exception (UDP and TCP 53). Metadata: ipBlock with except: [169.254.169.254/32]. Ports are container ports.
  • Docs: kubernetes.io “Network Policies”, docs.cilium.io “Layer 3 Examples”, “Layer 7 Examples”, “DNS based”.

Enterprise Note

  • Default deny and the DNS exception belong in the namespace template (GitOps), not in manual work.
  • Additional metadata protection in the cloud: IMDSv2 with hop limit 1 (AWS) or Workload Identity (GKE, AKS), so that Pods never inherit the node role in the first place.
  • Cluster-wide rules (block metadata, allow monitoring) via CiliumClusterwideNetworkPolicy, centrally by the platform team.
  • Hubble shows dropped flows with policy names – without visibility, default deny turns troubleshooting into a search in the dark.
  • toFQDNs needs Cilium’s DNS proxy; its availability thereby becomes business-critical.