Skip to content
Hands-on IT IT Drills

Day 2: DNS, Egress and Pitfalls

Goal: Lock down egress cleanly without losing DNS, the API server or ports, and trigger the typical pitfalls yourself. · Duration: approx. 90 min · Prerequisite: Day 1

Briefing

  • Default deny for egress also blocks DNS. Almost every “broken” policy is really a DNS problem.
  • Policies see Pod IPs and container ports, not Service IPs and Service ports.
  • A connection between two locked-down Pods needs two permissions: egress at the sender, ingress at the receiver.
  • ipBlock is meant for targets outside the cluster. For Pods, use selectors.
  • Every CNI treats host traffic (kubelet probes, hostNetwork Pods) a little differently.
 frontend ──DNS──► kube-dns (kube-system) :53 UDP/TCP     needs egress
    │
    └── api.shop:80 ──Service DNAT──► api Pod :8080        policy sees :8080
                                       needs ingress

Drill 1: Deny all – and DNS is gone

cd ~/guard-lab/tag02
alias cc=~/guard-lab/code/demo/conncheck.sh
kubectl -n shop get networkpolicy
cat > default-deny-all.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
EOF
kubectl apply -f default-deny-all.yaml
cc shop/frontend http://api.tenant-a
kubectl -n shop exec deploy/frontend -- nslookup api.tenant-a.svc.cluster.local

Expected:

No resources found in shop namespace.
networkpolicy.networking.k8s.io/default-deny-all created
✘ blockiert shop/frontend -> http://api.tenant-a (Timeout)
;; connection timed out; no servers could be reached

command terminated with exit code 1

The name can no longer be resolved. The packet to CoreDNS (10.96.0.10:53) is egress, and egress is now completely closed. Cross-check with the Pod IP:

IP=$(kubectl -n tenant-a get pod -l app=api -o jsonpath='{.items[0].status.podIP}')
cc shop/frontend http://$IP:8080

Expected:

✘ blockiert shop/frontend -> http://10.244.0.8:8080 (Timeout)

Even without DNS there is no way out. Default deny works in both directions.

Save your progress? Sign up for free

Drill 2: Allow DNS

cat > allow-dns.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
EOF
kubectl apply -f allow-dns.yaml
kubectl -n shop exec deploy/frontend -- nslookup api.tenant-a.svc.cluster.local
cc shop/frontend http://api.shop
cc shop/frontend http://nix.shop.svc.cluster.local

Expected (abridged):

networkpolicy.networking.k8s.io/allow-dns created
Server:     10.96.0.10
Name:   api.tenant-a.svc.cluster.local
Address: 10.96.4.91
✘ blockiert shop/frontend -> http://api.shop (Timeout)
✘ blockiert shop/frontend -> http://nix.shop.svc.cluster.local (DNS-Fehler)

DNS works again, as the DNS error for the nonexistent name shows: The answer “does not exist” arrives. api.shop is resolved, but the connection itself is not allowed yet. Both ports, UDP and TCP: Large responses (DNSSEC, many records) fall back to TCP.

Warning: The namespace shop and the real top-level domain .shop clash. Busybox nslookup ignores the search list from /etc/resolv.conf: kubectl -n shop exec deploy/frontend -- nslookup api.shop returns a public IP (in our case 185.180.19.198). curl and getent hosts api.shop use the search list and correctly end up at api.shop.svc.cluster.local. So in policies and configs always use the full name.

Save your progress? Sign up for free

Drill 3: Service port or container port?

The frontend should talk to the api. First idea: egress on port 80, as stated in the Service.

cat > frontend-to-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-to-api
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - port: 80
EOF
kubectl apply -f frontend-to-api.yaml
cc shop/frontend http://api.shop

Expected:

networkpolicy.networking.k8s.io/frontend-to-api created
✘ blockiert shop/frontend -> http://api.shop (Timeout)

Port 80 only exists on the Service. After the DNAT by kube-proxy, the packet goes to Pod IP:8080. So 8080:

sed -i.bak 's/    - port: 80/    - port: 8080/' frontend-to-api.yaml
kubectl apply -f frontend-to-api.yaml
cc shop/frontend http://api.shop

Expected:

networkpolicy.networking.k8s.io/frontend-to-api configured
✘ blockiert shop/frontend -> http://api.shop (Timeout)

Still closed. Egress at the frontend is open now, but the api has default deny for ingress. The other side is missing:

cat > api-from-frontend.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-frontend
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - port: http
EOF
kubectl apply -f api-from-frontend.yaml
cc shop/frontend http://api.shop
cc shop/frontend http://api.shop:9113/metrics

Expected:

networkpolicy.networking.k8s.io/api-from-frontend created
✔ erlaubt   shop/frontend -> http://api.shop (HTTP 200)
✘ blockiert shop/frontend -> http://api.shop:9113/metrics (Timeout)

Rule of thumb: With deny all in both directions, every connection needs two policies, one per end. Named ports (http) protect you from the 80/8080 trap because they refer directly to the container port.

Save your progress? Sign up for free

Drill 4: Port ranges with endPort

Some services use entire port ranges. endPort has been stable since Kubernetes 1.25:

cat > api-range.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-range
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 9000
      endPort: 9200
EOF
cat > api-metrics-in.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-metrics-in
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}
    ports:
    - port: metrics
EOF
kubectl apply -f api-range.yaml -f api-metrics-in.yaml
cc shop/frontend http://api.shop:9113/metrics

Expected:

networkpolicy.networking.k8s.io/api-range created
networkpolicy.networking.k8s.io/api-metrics-in created
✔ erlaubt   shop/frontend -> http://api.shop:9113/metrics (HTTP 200)

And the combination of a named port and a range?

cat > bad-range.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: bad-range
  namespace: shop
spec:
  podSelector: {}
  egress:
  - ports:
    - port: metrics
      endPort: 9200
EOF
kubectl apply -f bad-range.yaml

Expected (wrapped):

The NetworkPolicy "bad-range" is invalid: spec.egress[0].ports[0].endPort: Invalid
value: 9200: may not be specified when `port` is non-numeric

Good: API validation catches the mistake. Whether the CNI actually implements endPort, you still always check with a test. Older CNIs silently ignored the field.

Save your progress? Sign up for free

Drill 5: The API server – the ipBlock trap

Operators, sidecars and controllers talk to the API server. Under deny all, that no longer works:

cc shop/api https://kubernetes.default.svc/version
kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}{"\n"}'
kubectl get endpointslice -l kubernetes.io/service-name=kubernetes \
  -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}{"\n"}'

Expected:

✘ blockiert shop/api -> https://kubernetes.default.svc/version (Timeout)
10.96.0.1
172.18.0.8:6443

The obvious idea would be an ipBlock with the Service IP 10.96.0.1/32 and port 443:

cat > api-to-apiserver.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-to-apiserver
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.96.0.1/32
    ports:
    - port: 443
EOF
kubectl apply -f api-to-apiserver.yaml
cc shop/api https://kubernetes.default.svc/version

Expected:

networkpolicy.networking.k8s.io/api-to-apiserver created
✘ blockiert shop/api -> https://kubernetes.default.svc/version (Timeout)

The same DNAT trap as in Drill 3, just with IPs: The policy sees the real address of the API server and its port 6443.

EP=$(kubectl get endpointslice -l kubernetes.io/service-name=kubernetes \
  -o jsonpath='{.items[0].endpoints[0].addresses[0]}')
sed -i.bak -e "s#10.96.0.1/32#$EP/32#" -e 's/port: 443/port: 6443/' \
  api-to-apiserver.yaml
kubectl apply -f api-to-apiserver.yaml
cc shop/api https://kubernetes.default.svc/version

Expected:

networkpolicy.networking.k8s.io/api-to-apiserver configured
✔ erlaubt   shop/api -> https://kubernetes.default.svc/version (HTTP 200)

Warning: The endpoint IPs change with every new control plane node. With multiple control plane nodes you have to list all of them. Cilium solves this more elegantly with toEntities: [kube-apiserver] (Day 4).

Save your progress? Sign up for free

Drill 6: Internet yes, cluster no – ipBlock with except

The api should reach external HTTPS services, but not its neighbors in the cluster:

cc shop/api https://example.com
cat > api-to-internet.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-to-internet
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
    ports:
    - port: 443
    - port: 8080
EOF
kubectl apply -f api-to-internet.yaml
cc shop/api https://example.com
cc shop/api http://api.tenant-a
IP=$(kubectl -n tenant-a get pod -l app=api -o jsonpath='{.items[0].status.podIP}')
cc shop/api http://$IP:8080

Expected:

✘ blockiert shop/api -> https://example.com (Timeout)
networkpolicy.networking.k8s.io/api-to-internet created
✔ erlaubt   shop/api -> https://example.com (HTTP 200)
✘ blockiert shop/api -> http://api.tenant-a (Timeout)
✘ blockiert shop/api -> http://10.244.0.8:8080 (Timeout)

Port 8080 is open in the policy, yet the api still cannot reach tenant-a: The Pod IP 10.244.0.8 lies in 10.0.0.0/8 and thus in except. Without except, 0.0.0.0/0 would be a free pass into the entire cluster.

Warning: Whether ipBlock applies to Pod IPs at all is left open by the specification. For cluster targets, never rely on ipBlock; rely on selectors. And in clouds, Pod and node networks often lie in ranges you do not want to exclude.

Save your progress? Sign up for free

Drill 7: Probes and hostNetwork

Under deny all, no Pod has ingress permission from the node. Do the readiness probes survive?

kubectl -n shop delete pod -l app=api --wait=false
sleep 25
kubectl -n shop get pods -l app=api

Expected:

pod "api-5ccd4587ff-gwhbn" deleted from shop namespace
NAME                   READY   STATUS    RESTARTS   AGE
api-5ccd4587ff-m5966   1/1     Running   0          25s

The new Pod becomes Ready. kindnet lets the kubelet’s probes through, as do Calico and Cilium in their default settings. Now a Pod in the node’s host network:

cat > hostnet.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: hostnet
  namespace: default
  labels: {app: hostnet}
spec:
  hostNetwork: true
  terminationGracePeriodSeconds: 1
  containers:
  - name: curl
    image: curlimages/curl:8.22.0
    command: [sleep, "3600"]
EOF
kubectl apply -f hostnet.yaml
kubectl wait --for=condition=Ready pod/hostnet --timeout=60s
kubectl get pod hostnet -o wide | awk '{print $1, $6}'
IP=$(kubectl -n shop get pod -l app=api -o jsonpath='{.items[0].status.podIP}')
cc default/hostnet http://$IP:8080

Expected:

pod/hostnet created
pod/hostnet condition met
NAME IP
hostnet 172.18.0.8
✘ blockiert default/hostnet -> http://10.244.0.13:8080 (Timeout)

A hostNetwork Pod has the node’s IP and no Pod label of its own in the packet. In our kindnet test it is blocked, but the kubelet probes from the same node are not. Other CNIs decide differently here, because the specification deliberately leaves host traffic open. Test it yourself on every target system.

kubectl delete pod hostnet

Security: hostNetwork: true bypasses the Pod identity. Forbid it via Pod Security or Kyverno for all app namespaces (Day 8).

Save your progress? Sign up for free

Checkpoint

cc=~/guard-lab/code/demo/conncheck.sh
while read -r src url want; do
  "$cc" "$src" "$url" "$want" >/dev/null \
    && echo "✔ $src -> $url ($want)" || echo "✘ $src -> $url (erwartet: $want)"
done <<'EOF'
shop/frontend http://api.shop allow
shop/frontend http://api.shop:9113/metrics allow
shop/frontend http://api.tenant-a deny
shop/api https://kubernetes.default.svc/version allow
shop/api https://example.com allow
shop/api http://api.tenant-a deny
shop/db http://api.shop deny
EOF
kubectl -n shop get netpol allow-dns >/dev/null 2>&1 \
  && echo "✔ allow-dns" || echo "✘ allow-dns fehlt"

Challenge

The client should be able to fetch http://frontend.shop/api/orders again, but still not reach the api directly. The frontend resolves api.shop at runtime via the nginx resolver.

Hint: Exactly one policy is missing. You already have egress from the frontend to the api and DNS.

Solution

cat > frontend-from-all.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-from-all
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector: {}
    ports:
    - port: http
EOF
kubectl apply -f frontend-from-all.yaml
kubectl exec client -- curl -s -m 5 http://frontend.shop/api/orders
cc default/client http://api.shop deny

Expected (abridged, may differ depending on version):

networkpolicy.networking.k8s.io/frontend-from-all created
{"svc":"api","ns":"shop","path":"/api/orders"}
✘ blockiert default/client -> http://api.shop (Timeout)

Without allow-dns, nginx returns a 5xx error after about 2 s: The connection to the frontend is established, but the name api.shop.svc.cluster.local cannot be resolved.

Clean up for Day 3:

kubectl -n shop delete networkpolicy --all

Save your progress? Sign up for free

Enterprise Note

  • Always roll out default deny for egress together with an allow-dns policy. Otherwise every namespace breaks immediately.
  • Maintain the permission to the API server centrally (AdminNetworkPolicy or CiliumClusterwideNetworkPolicy, Day 5), not in every team namespace with fixed IPs.
  • External targets by IP are fragile. For SaaS endpoints with changing IPs you need FQDN policies (Cilium toFQDNs, Day 4) or an egress proxy.
  • At a customer site, check how the CNI handles host traffic, node ports and hostNetwork. OpenShift (OVN-Kubernetes) and RKE2 (Canal) do not behave identically.
  • Use fully qualified DNS names (svc.cluster.local) in app configs and, if needed, set dnsConfig.options: ndots to avoid unnecessary upstream queries.