Day 2: NetworkPolicies and CiliumNetworkPolicy
Goal: You block Pod traffic by default, open only the paths you need, keep Pods away from node endpoints and cloud metadata, and use Cilium for L7 and DNS rules. · Duration: approx. 90 min · Prerequisite: Day 1
Briefing
- CKS: Cluster Setup (15 %) – “Use network security policies to restrict cluster level access” and “Protect node metadata and endpoints”.
- Without a policy, every Pod talks to every other Pod – and to the
kubelet, the API server and the metadata IP
169.254.169.254, through which cloud VMs hand out credentials. - Policies are additive. The standard NetworkPolicy has no “deny”; Cilium does.
- Cilium treats nodes as an identity of their own:
ipBlockdoes not cover them (Drill 4).
tag02-other/outsider ──┐
tag02/client ──────────┴─> tag02/web :80
└──X──> 169.254.169.254 · Nodes :10250/:6443 · Internet (targeted only)
Drill 1: The test field
mkdir -p ~/cks/tag02 && cd ~/cks/tag02
k create ns tag02
k create ns tag02-other
k config set-context --current --namespace=tag02
k create deployment web --image=nginx:1.29 --replicas=2 --port=80
k expose deployment web --port=80
k run client --image=busybox:1.37 -- sleep 3600
k run outsider -n tag02-other --image=busybox:1.37 -- sleep 3600
k rollout status deployment web
k exec client -- wget -qO- -T2 http://web | grep title
k exec -n tag02-other outsider -- wget -qO- -T2 http://web.tag02 | grep title
k exec client -- nc -zv -w2 172.16.0.228 10250
k exec client -- nc -zv -w2 172.16.0.126 6443
k exec client -- nc -zv -w2 169.254.169.254 80Expected (shortened):
deployment "web" successfully rolled out
<title>Welcome to nginx!</title>
<title>Welcome to nginx!</title>
172.16.0.228 (172.16.0.228:10250) open
172.16.0.126 (172.16.0.126:6443) open
nc: 169.254.169.254 (169.254.169.254:80): Network is unreachable
command terminated with exit code 1
Every Pod reaches the kubelet and API server of the nodes. There is
no metadata service in the lab, hence
Network is unreachable – in AWS, Azure or GCP you would get
an answer at this point, often with credentials of the node role. Always
set timeouts (-T2, -w2), otherwise you wait
forever when traffic is blocked.
Save your progress? Sign up for free
Drill 2: Default deny in both directions
cat > default-deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
EOF
k apply -f default-deny.yaml
k exec client -- wget -qO- -T2 http://web
k exec -n tag02-other outsider -- wget -qO- -T2 http://web.tag02
k exec client -- nc -zv -w2 172.16.0.228 10250
k exec client -- nc -zv -w2 169.254.169.254 80Expected:
networkpolicy.networking.k8s.io/default-deny created
wget: bad address 'web'
command terminated with exit code 1
wget: download timed out
command terminated with exit code 1
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
nc: 169.254.169.254 (169.254.169.254:80): Connection timed out
command terminated with exit code 1
podSelector: {} selects all Pods in the namespace;
without ingress/egress lists, nothing is
allowed. bad address shows: DNS is egress too.
Network is unreachable turns into timed out –
Cilium already drops the packets at the Pod interface, before they leave
the node.
Save your progress? Sign up for free
Drill 3: Allow DNS and targeted paths
cat > allow-dns.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
EOF
k apply -f allow-dns.yaml
k exec client -- nslookup web.tag02.svc.cluster.local
k exec client -- wget -qO- -T2 http://webExpected:
networkpolicy.networking.k8s.io/allow-dns created
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: web.tag02.svc.cluster.local
Address: 10.99.244.41
wget: download timed out
command terminated with exit code 1
The name resolves, the connection stays closed. Inside a namespace
with default deny, every path needs two permissions: egress at the
sender, ingress at the receiver. On top, the namespace
tag02-other is allowed in:
cat > web-policies.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-ingress
spec:
podSelector:
matchLabels:
app: web
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
run: client
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: tag02-other
ports:
- port: 80
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: client-egress
spec:
podSelector:
matchLabels:
run: client
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: web
ports:
- port: 80
EOF
k apply -f web-policies.yaml
k exec client -- wget -qO- -T2 http://web | grep title
k exec -n tag02-other outsider -- wget -qO- -T2 http://web.tag02 | grep title
k describe netpol web-ingress | sed -n '/Allowing ingress/,/Not affecting/p'Expected:
networkpolicy.networking.k8s.io/web-ingress created
networkpolicy.networking.k8s.io/client-egress created
<title>Welcome to nginx!</title>
<title>Welcome to nginx!</title>
Allowing ingress traffic:
To Port: 80/TCP
From:
PodSelector: run=client
From:
NamespaceSelector: kubernetes.io/metadata.name=tag02-other
Not affecting egress traffic
Two From: blocks mean OR: Pods with
run=client from tag02 or any
Pod from tag02-other. k run sets
run: client automatically.
Save your progress? Sign up for free
Drill 4: Internet yes, metadata no – ipBlock with except
client should reach the internet, but never the metadata
IP:
cat > client-world.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: client-world
spec:
podSelector:
matchLabels:
run: client
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32
EOF
k apply -f client-world.yaml
k exec client -- nc -zv -w3 kubernetes.io 443
k exec client -- nc -zv -w2 169.254.169.254 80
k exec client -- nc -zv -w2 172.16.0.228 10250
k exec client -- nc -zv -w2 172.16.0.126 6443Expected:
networkpolicy.networking.k8s.io/client-world created
kubernetes.io (3.33.186.135:443) open
nc: 169.254.169.254 (169.254.169.254:80): Connection timed out
command terminated with exit code 1
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
nc: 172.16.0.126 (172.16.0.126:6443): Connection timed out
command terminated with exit code 1
Surprise: the nodes are blocked, even though 0.0.0.0/0
includes them. Cross-check with exactly the node network:
k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tmp-nodes
spec:
podSelector:
matchLabels:
run: client
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 172.16.0.0/24
EOF
k exec client -- nc -zv -w2 172.16.0.228 10250
k delete netpol tmp-nodesExpected:
networkpolicy.networking.k8s.io/tmp-nodes created
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
networkpolicy.networking.k8s.io "tmp-nodes" deleted from tag02 namespace
Cilium maps node IPs to the identities host and
remote-node, and by default ipBlock only
covers “world”. Other CNIs (Calico, for example) let
0.0.0.0/0 reach the nodes as well. That is why, in the
exam: write the node network and the metadata IP explicitly into
except – that works with every CNI.
Save your progress? Sign up for free
Drill 5: CiliumNetworkPolicy – L3/L4 and entities
Cilium policies in a namespace of their own, so that nothing mixes with Drills 2 to 4:
k create ns tag02-cilium
k -n tag02-cilium create deployment web --image=nginx:1.29 --port=80
k -n tag02-cilium expose deployment web --port=80
k -n tag02-cilium run client --image=busybox:1.37 -- sleep 3600
k -n tag02-cilium rollout status deployment web
cat > cnp-client.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: client-egress
namespace: tag02-cilium
spec:
endpointSelector:
matchLabels:
run: client
egress:
- toEndpoints:
- matchLabels:
app: web
toPorts:
- ports:
- port: "80"
protocol: TCP
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
- toEntities:
- kube-apiserver
toPorts:
- ports:
- port: "6443"
protocol: TCP
EOF
k apply -f cnp-client.yaml
k get cnp -n tag02-ciliumExpected:
ciliumnetworkpolicy.cilium.io/client-egress created
NAME AGE VALID
client-egress 3s True
Always check VALID True: Cilium accepts faulty policies
but does not enforce them. Ports are quoted. Now test:
k -n tag02-cilium exec client -- wget -qO- -T2 http://web | grep title
k -n tag02-cilium exec client -- nc -zv -w2 kubernetes.default 443
k -n tag02-cilium exec client -- nc -zv -w2 172.16.0.228 10250
k -n tag02-cilium exec client -- nc -zv -w2 kubernetes.io 443Expected:
<title>Welcome to nginx!</title>
kubernetes.default (10.96.0.1:443) open
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
nc: kubernetes.io (3.33.186.135:443): Connection timed out
command terminated with exit code 1
A CNP egress section with rules switches the Pod to default deny.
toEntities: kube-apiserver allows exactly the API server –
no matter which node it runs on; the kubelet on worker1 stays closed.
More entities: world, cluster,
host, remote-node, all
(k explain cnp.spec.egress.toEntities).
Save your progress? Sign up for free
Drill 6: L7 rule for HTTP
Standard NetworkPolicies only know ports. Cilium also checks method and path:
k -n tag02-cilium exec deploy/web -- \
sh -c 'echo geheim > /usr/share/nginx/html/admin.html'
k -n tag02-cilium exec client -- wget -qO- -T2 http://web/admin.html
cat > cnp-web-l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: web-l7
namespace: tag02-cilium
spec:
endpointSelector:
matchLabels:
app: web
ingress:
- fromEndpoints:
- matchLabels:
run: client
toPorts:
- ports:
- port: "80"
protocol: TCP
rules:
http:
- method: GET
path: "/"
- method: GET
path: "/index.html"
EOF
k apply -f cnp-web-l7.yaml
k -n tag02-cilium exec client -- wget -qO- -T2 http://web | grep title
k -n tag02-cilium exec client -- wget -qO- -T2 http://web/admin.html
k -n tag02-cilium exec client -- wget -qO- -T2 --post-data=x http://web/Expected:
geheim
ciliumnetworkpolicy.cilium.io/web-l7 created
<title>Welcome to nginx!</title>
wget: server returned error: HTTP/1.1 403 Forbidden
command terminated with exit code 1
wget: server returned error: HTTP/1.1 403 Forbidden
command terminated with exit code 1
The 403 does not come from nginx but from Cilium’s Envoy proxy
(DaemonSet cilium-envoy). path is a regular
expression over the whole path: / does not allow
/admin.html.
Save your progress? Sign up for free
Drill 7: toFQDNs – egress by name
External destinations change their IPs. Cilium learns them from the
DNS responses – for that, the DNS rule must go through the Cilium DNS
proxy (rules: dns):
cat > cnp-fqdn.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: client-fqdn
namespace: tag02-cilium
spec:
endpointSelector:
matchLabels:
run: client
egress:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
- toFQDNs:
- matchName: kubernetes.io
toPorts:
- ports:
- port: "443"
protocol: TCP
EOF
k apply -f cnp-fqdn.yaml
k -n tag02-cilium exec client -- nc -zv -w3 kubernetes.io 443
k -n tag02-cilium exec client -- nc -zv -w3 kubernetes.io 80
k -n tag02-cilium exec client -- nc -zv -w3 github.com 443Expected:
ciliumnetworkpolicy.cilium.io/client-fqdn created
kubernetes.io (3.33.186.135:443) open
nc: kubernetes.io (3.33.186.135:80): Connection timed out
command terminated with exit code 1
nc: github.com (140.82.121.3:443): Connection timed out
command terminated with exit code 1
Both CNPs select client and add up: web, DNS, the API
server and kubernetes.io:443 are open, nothing else.
Without the dns rule, toFQDNs would have no
effect, because Cilium never sees the name resolution.
Save your progress? Sign up for free
Drill 8: Block metadata cluster-wide – deny rule
A real “deny” for all Pods, without sending them into default deny.
For practice only for the namespace tag02-cilium (in
production endpointSelector: {}):
k -n tag02-cilium run probe --image=busybox:1.37 -- sleep 3600
k -n tag02-cilium exec probe -- nc -zv -w2 169.254.169.254 80
cat > ccnp-metadata.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: tag02-deny-metadata
spec:
endpointSelector:
matchLabels:
k8s:io.kubernetes.pod.namespace: tag02-cilium
enableDefaultDeny:
egress: false
ingress: false
egressDeny:
- toCIDR:
- 169.254.169.254/32
EOF
k apply -f ccnp-metadata.yaml
k get ccnp
k -n tag02-cilium exec probe -- nc -zv -w2 169.254.169.254 80
k -n tag02-cilium exec probe -- nc -zv -w3 kubernetes.io 443Expected:
nc: 169.254.169.254 (169.254.169.254:80): Network is unreachable
command terminated with exit code 1
ciliumclusterwidenetworkpolicy.cilium.io/tag02-deny-metadata created
NAME VALID
tag02-deny-metadata True
nc: 169.254.169.254 (169.254.169.254:80): Connection timed out
command terminated with exit code 1
kubernetes.io (15.197.167.90:443) open
probe still reaches the internet; only the metadata IP
is gone. enableDefaultDeny: false prevents the policy from
blocking everything else. Deny rules take precedence over every allow –
not even a later allow policy opens the metadata.
Save your progress? Sign up for free
Exercise: Speed Drills
- Block all incoming traffic in
tag02-other. Target time: 45 s - Show all Cilium policies in the cluster, namespaced and cluster-wide, with validity. Target time: 15 s
- Extend
web-l7:clientmay additionally read all paths under/adminvia GET. Target time: 60 s - Which policy types and which exceptions does
client-worldhave? Target time: 30 s
Solution
k apply -n tag02-other -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
spec:
podSelector: {}
policyTypes:
- Ingress
EOF
k get cnp,ccnp -A
vim cnp-web-l7.yamlAdd below the entry for /index.html:
- method: GET
path: "/admin.*"k apply -f cnp-web-l7.yaml
k -n tag02-cilium exec client -- wget -qO- -T2 http://web/admin.html
k get netpol client-world \
-o jsonpath='{.spec.policyTypes} {.spec.egress[0].to[0].ipBlock.except}{"\n"}'Expected (shortened):
networkpolicy.networking.k8s.io/default-deny-ingress created
NAMESPACE NAME AGE VALID
tag02-cilium ciliumnetworkpolicy.cilium.io/client-egress 98s True
tag02-cilium ciliumnetworkpolicy.cilium.io/client-fqdn 68s True
tag02-cilium ciliumnetworkpolicy.cilium.io/web-l7 81s True
NAMESPACE NAME VALID
ciliumclusterwidenetworkpolicy.cilium.io/tag02-deny-metadata True
ciliumnetworkpolicy.cilium.io/web-l7 configured
geheim
["Egress"] ["169.254.169.254/32"]
Save your progress? Sign up for free
Checkpoint
k get netpol -n tag02 -o custom-columns=NAME:.metadata.name,TYPES:.spec.policyTypesExpected:
NAME TYPES
allow-dns [Egress]
client-egress [Egress]
client-world [Egress]
default-deny [Ingress Egress]
web-ingress [Ingress]
k exec -n tag02 client -- nc -zv -w2 169.254.169.254 80Expected: Connection timed out
k -n tag02-cilium exec client -- wget -qO- -T2 http://web/geheim.htmlExpected:
wget: server returned error: HTTP/1.1 403 Forbidden – the
L7 rule takes effect.
k get ccnp tag02-deny-metadataExpected: VALID = True
Challenge
client in tag02 should additionally reach
the API server via kubernetes.default:443, but still no
kubelet. A NetworkPolicy with ipBlock: 172.16.0.126/32 and
port 6443 does not help here – find the way that works with Cilium.
Hint: Drill 4 explains why ipBlock
fails; Drill 5 shows the entity.
Solution
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: client-api
spec:
endpointSelector:
matchLabels:
run: client
egress:
- toEntities:
- kube-apiserver
toPorts:
- ports:
- port: "6443"
protocol: TCP
EOF
k exec client -- nc -zv -w2 kubernetes.default 443
k exec client -- nc -zv -w2 172.16.0.228 10250Expected:
ciliumnetworkpolicy.cilium.io/client-api created
kubernetes.default (10.96.0.1:443) open
nc: 172.16.0.228 (172.16.0.228:10250): Connection timed out
command terminated with exit code 1
The port is 6443, not 443: policies take effect after the Service translation. Clean up:
k config set-context --current --namespace=default
k delete ns tag02 tag02-other tag02-cilium
k delete ccnp tag02-deny-metadataSave your progress? Sign up for free
Exam Tip
- First find out which CNI is running
(
k get pods -n kube-system). Use Cilium objects only when the task asks for them; otherwise the standard NetworkPolicy. - Always pair egress blocks with a DNS exception (UDP and TCP 53).
Metadata:
ipBlockwithexcept: [169.254.169.254/32]. Ports are container ports. - Docs: kubernetes.io “Network Policies”, docs.cilium.io “Layer 3 Examples”, “Layer 7 Examples”, “DNS based”.
Enterprise Note
- Default deny and the DNS exception belong in the namespace template (GitOps), not in manual work.
- Additional metadata protection in the cloud: IMDSv2 with hop limit 1 (AWS) or Workload Identity (GKE, AKS), so that Pods never inherit the node role in the first place.
- Cluster-wide rules (block metadata, allow monitoring) via
CiliumClusterwideNetworkPolicy, centrally by the platform team. - Hubble shows dropped flows with policy names – without visibility, default deny turns troubleshooting into a search in the dark.
toFQDNsneeds Cilium’s DNS proxy; its availability thereby becomes business-critical.