Day 2: DNS, Egress and Pitfalls
Goal: Lock down egress cleanly without losing DNS, the API server or ports, and trigger the typical pitfalls yourself. · Duration: approx. 90 min · Prerequisite: Day 1
Briefing
- Default deny for egress also blocks DNS. Almost every “broken” policy is really a DNS problem.
- Policies see Pod IPs and container ports, not Service IPs and Service ports.
- A connection between two locked-down Pods needs two permissions: egress at the sender, ingress at the receiver.
ipBlockis meant for targets outside the cluster. For Pods, use selectors.- Every CNI treats host traffic (kubelet probes,
hostNetworkPods) a little differently.
frontend ──DNS──► kube-dns (kube-system) :53 UDP/TCP needs egress
│
└── api.shop:80 ──Service DNAT──► api Pod :8080 policy sees :8080
needs ingress
Drill 1: Deny all – and DNS is gone
cd ~/guard-lab/tag02
alias cc=~/guard-lab/code/demo/conncheck.sh
kubectl -n shop get networkpolicy
cat > default-deny-all.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: shop
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
EOF
kubectl apply -f default-deny-all.yaml
cc shop/frontend http://api.tenant-a
kubectl -n shop exec deploy/frontend -- nslookup api.tenant-a.svc.cluster.localExpected:
No resources found in shop namespace.
networkpolicy.networking.k8s.io/default-deny-all created
✘ blockiert shop/frontend -> http://api.tenant-a (Timeout)
;; connection timed out; no servers could be reached
command terminated with exit code 1
The name can no longer be resolved. The packet to CoreDNS
(10.96.0.10:53) is egress, and egress is now completely
closed. Cross-check with the Pod IP:
IP=$(kubectl -n tenant-a get pod -l app=api -o jsonpath='{.items[0].status.podIP}')
cc shop/frontend http://$IP:8080Expected:
✘ blockiert shop/frontend -> http://10.244.0.8:8080 (Timeout)
Even without DNS there is no way out. Default deny works in both directions.
Save your progress? Sign up for free
Drill 2: Allow DNS
cat > allow-dns.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: shop
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
EOF
kubectl apply -f allow-dns.yaml
kubectl -n shop exec deploy/frontend -- nslookup api.tenant-a.svc.cluster.local
cc shop/frontend http://api.shop
cc shop/frontend http://nix.shop.svc.cluster.localExpected (abridged):
networkpolicy.networking.k8s.io/allow-dns created
Server: 10.96.0.10
Name: api.tenant-a.svc.cluster.local
Address: 10.96.4.91
✘ blockiert shop/frontend -> http://api.shop (Timeout)
✘ blockiert shop/frontend -> http://nix.shop.svc.cluster.local (DNS-Fehler)
DNS works again, as the DNS error for the nonexistent name shows: The
answer “does not exist” arrives. api.shop is resolved, but
the connection itself is not allowed yet. Both ports, UDP and TCP: Large
responses (DNSSEC, many records) fall back to TCP.
Warning: The namespace
shopand the real top-level domain.shopclash. Busyboxnslookupignores the search list from/etc/resolv.conf:kubectl -n shop exec deploy/frontend -- nslookup api.shopreturns a public IP (in our case185.180.19.198).curlandgetent hosts api.shopuse the search list and correctly end up atapi.shop.svc.cluster.local. So in policies and configs always use the full name.
Save your progress? Sign up for free
Drill 3: Service port or container port?
The frontend should talk to the api. First idea: egress on port 80, as stated in the Service.
cat > frontend-to-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: frontend-to-api
namespace: shop
spec:
podSelector:
matchLabels:
app: frontend
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: api
ports:
- port: 80
EOF
kubectl apply -f frontend-to-api.yaml
cc shop/frontend http://api.shopExpected:
networkpolicy.networking.k8s.io/frontend-to-api created
✘ blockiert shop/frontend -> http://api.shop (Timeout)
Port 80 only exists on the Service. After the DNAT by kube-proxy, the packet goes to Pod IP:8080. So 8080:
sed -i.bak 's/ - port: 80/ - port: 8080/' frontend-to-api.yaml
kubectl apply -f frontend-to-api.yaml
cc shop/frontend http://api.shopExpected:
networkpolicy.networking.k8s.io/frontend-to-api configured
✘ blockiert shop/frontend -> http://api.shop (Timeout)
Still closed. Egress at the frontend is open now, but the api has default deny for ingress. The other side is missing:
cat > api-from-frontend.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-from-frontend
namespace: shop
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- port: http
EOF
kubectl apply -f api-from-frontend.yaml
cc shop/frontend http://api.shop
cc shop/frontend http://api.shop:9113/metricsExpected:
networkpolicy.networking.k8s.io/api-from-frontend created
✔ erlaubt shop/frontend -> http://api.shop (HTTP 200)
✘ blockiert shop/frontend -> http://api.shop:9113/metrics (Timeout)
Rule of thumb: With deny all in both directions, every connection
needs two policies, one per end. Named ports (http) protect
you from the 80/8080 trap because they refer directly to the container
port.
Save your progress? Sign up for free
Drill 4: Port ranges with endPort
Some services use entire port ranges. endPort has been
stable since Kubernetes 1.25:
cat > api-range.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-range
namespace: shop
spec:
podSelector:
matchLabels:
app: frontend
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: api
ports:
- protocol: TCP
port: 9000
endPort: 9200
EOF
cat > api-metrics-in.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-metrics-in
namespace: shop
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {}
ports:
- port: metrics
EOF
kubectl apply -f api-range.yaml -f api-metrics-in.yaml
cc shop/frontend http://api.shop:9113/metricsExpected:
networkpolicy.networking.k8s.io/api-range created
networkpolicy.networking.k8s.io/api-metrics-in created
✔ erlaubt shop/frontend -> http://api.shop:9113/metrics (HTTP 200)
And the combination of a named port and a range?
cat > bad-range.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: bad-range
namespace: shop
spec:
podSelector: {}
egress:
- ports:
- port: metrics
endPort: 9200
EOF
kubectl apply -f bad-range.yamlExpected (wrapped):
The NetworkPolicy "bad-range" is invalid: spec.egress[0].ports[0].endPort: Invalid
value: 9200: may not be specified when `port` is non-numeric
Good: API validation catches the mistake. Whether the CNI actually
implements endPort, you still always check with a test.
Older CNIs silently ignored the field.
Save your progress? Sign up for free
Drill 5: The API server – the ipBlock trap
Operators, sidecars and controllers talk to the API server. Under deny all, that no longer works:
cc shop/api https://kubernetes.default.svc/version
kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}{"\n"}'
kubectl get endpointslice -l kubernetes.io/service-name=kubernetes \
-o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}{"\n"}'Expected:
✘ blockiert shop/api -> https://kubernetes.default.svc/version (Timeout)
10.96.0.1
172.18.0.8:6443
The obvious idea would be an ipBlock with the Service IP
10.96.0.1/32 and port 443:
cat > api-to-apiserver.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-to-apiserver
namespace: shop
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.96.0.1/32
ports:
- port: 443
EOF
kubectl apply -f api-to-apiserver.yaml
cc shop/api https://kubernetes.default.svc/versionExpected:
networkpolicy.networking.k8s.io/api-to-apiserver created
✘ blockiert shop/api -> https://kubernetes.default.svc/version (Timeout)
The same DNAT trap as in Drill 3, just with IPs: The policy sees the real address of the API server and its port 6443.
EP=$(kubectl get endpointslice -l kubernetes.io/service-name=kubernetes \
-o jsonpath='{.items[0].endpoints[0].addresses[0]}')
sed -i.bak -e "s#10.96.0.1/32#$EP/32#" -e 's/port: 443/port: 6443/' \
api-to-apiserver.yaml
kubectl apply -f api-to-apiserver.yaml
cc shop/api https://kubernetes.default.svc/versionExpected:
networkpolicy.networking.k8s.io/api-to-apiserver configured
✔ erlaubt shop/api -> https://kubernetes.default.svc/version (HTTP 200)
Warning: The endpoint IPs change with every new control plane node. With multiple control plane nodes you have to list all of them. Cilium solves this more elegantly with
toEntities: [kube-apiserver](Day 4).
Save your progress? Sign up for free
Drill 6: Internet yes, cluster no – ipBlock with except
The api should reach external HTTPS services, but not its neighbors in the cluster:
cc shop/api https://example.com
cat > api-to-internet.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-to-internet
namespace: shop
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
- port: 8080
EOF
kubectl apply -f api-to-internet.yaml
cc shop/api https://example.com
cc shop/api http://api.tenant-a
IP=$(kubectl -n tenant-a get pod -l app=api -o jsonpath='{.items[0].status.podIP}')
cc shop/api http://$IP:8080Expected:
✘ blockiert shop/api -> https://example.com (Timeout)
networkpolicy.networking.k8s.io/api-to-internet created
✔ erlaubt shop/api -> https://example.com (HTTP 200)
✘ blockiert shop/api -> http://api.tenant-a (Timeout)
✘ blockiert shop/api -> http://10.244.0.8:8080 (Timeout)
Port 8080 is open in the policy, yet the api still cannot reach
tenant-a: The Pod IP 10.244.0.8 lies in
10.0.0.0/8 and thus in except. Without
except, 0.0.0.0/0 would be a free pass into
the entire cluster.
Warning: Whether
ipBlockapplies to Pod IPs at all is left open by the specification. For cluster targets, never rely onipBlock; rely on selectors. And in clouds, Pod and node networks often lie in ranges you do not want to exclude.
Save your progress? Sign up for free
Drill 7: Probes and hostNetwork
Under deny all, no Pod has ingress permission from the node. Do the readiness probes survive?
kubectl -n shop delete pod -l app=api --wait=false
sleep 25
kubectl -n shop get pods -l app=apiExpected:
pod "api-5ccd4587ff-gwhbn" deleted from shop namespace
NAME READY STATUS RESTARTS AGE
api-5ccd4587ff-m5966 1/1 Running 0 25s
The new Pod becomes Ready. kindnet lets the kubelet’s
probes through, as do Calico and Cilium in their default settings. Now a
Pod in the node’s host network:
cat > hostnet.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: hostnet
namespace: default
labels: {app: hostnet}
spec:
hostNetwork: true
terminationGracePeriodSeconds: 1
containers:
- name: curl
image: curlimages/curl:8.22.0
command: [sleep, "3600"]
EOF
kubectl apply -f hostnet.yaml
kubectl wait --for=condition=Ready pod/hostnet --timeout=60s
kubectl get pod hostnet -o wide | awk '{print $1, $6}'
IP=$(kubectl -n shop get pod -l app=api -o jsonpath='{.items[0].status.podIP}')
cc default/hostnet http://$IP:8080Expected:
pod/hostnet created
pod/hostnet condition met
NAME IP
hostnet 172.18.0.8
✘ blockiert default/hostnet -> http://10.244.0.13:8080 (Timeout)
A hostNetwork Pod has the node’s IP and no Pod label of
its own in the packet. In our kindnet test it is blocked, but the
kubelet probes from the same node are not. Other CNIs decide differently
here, because the specification deliberately leaves host traffic open.
Test it yourself on every target system.
kubectl delete pod hostnetSecurity:
hostNetwork: truebypasses the Pod identity. Forbid it via Pod Security or Kyverno for all app namespaces (Day 8).
Save your progress? Sign up for free
Checkpoint
cc=~/guard-lab/code/demo/conncheck.sh
while read -r src url want; do
"$cc" "$src" "$url" "$want" >/dev/null \
&& echo "✔ $src -> $url ($want)" || echo "✘ $src -> $url (erwartet: $want)"
done <<'EOF'
shop/frontend http://api.shop allow
shop/frontend http://api.shop:9113/metrics allow
shop/frontend http://api.tenant-a deny
shop/api https://kubernetes.default.svc/version allow
shop/api https://example.com allow
shop/api http://api.tenant-a deny
shop/db http://api.shop deny
EOF
kubectl -n shop get netpol allow-dns >/dev/null 2>&1 \
&& echo "✔ allow-dns" || echo "✘ allow-dns fehlt"Challenge
The client should be able to fetch
http://frontend.shop/api/orders again, but still not reach
the api directly. The frontend resolves api.shop at runtime
via the nginx resolver.
Hint: Exactly one policy is missing. You already have egress from the frontend to the api and DNS.
Solution
cat > frontend-from-all.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: frontend-from-all
namespace: shop
spec:
podSelector:
matchLabels:
app: frontend
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
ports:
- port: http
EOF
kubectl apply -f frontend-from-all.yaml
kubectl exec client -- curl -s -m 5 http://frontend.shop/api/orders
cc default/client http://api.shop denyExpected (abridged, may differ depending on version):
networkpolicy.networking.k8s.io/frontend-from-all created
{"svc":"api","ns":"shop","path":"/api/orders"}
✘ blockiert default/client -> http://api.shop (Timeout)
Without allow-dns, nginx returns a 5xx error after about
2 s: The connection to the frontend is established, but the name
api.shop.svc.cluster.local cannot be resolved.
Clean up for Day 3:
kubectl -n shop delete networkpolicy --allSave your progress? Sign up for free
Enterprise Note
- Always roll out default deny for egress together with an
allow-dnspolicy. Otherwise every namespace breaks immediately. - Maintain the permission to the API server centrally (AdminNetworkPolicy or CiliumClusterwideNetworkPolicy, Day 5), not in every team namespace with fixed IPs.
- External targets by IP are fragile. For SaaS endpoints with changing
IPs you need FQDN policies (Cilium
toFQDNs, Day 4) or an egress proxy. - At a customer site, check how the CNI handles host traffic, node
ports and
hostNetwork. OpenShift (OVN-Kubernetes) and RKE2 (Canal) do not behave identically. - Use fully qualified DNS names (
svc.cluster.local) in app configs and, if needed, setdnsConfig.options: ndotsto avoid unnecessary upstream queries.