Skip to content
Hands-on IT IT Drills

Day 8: DNS and NetworkPolicies

Goal: You tell name errors apart from connection errors, watch CoreDNS at work and find NetworkPolicies that block too much or nothing at all. · Duration: approx. 75 min · Prerequisite: Day 7

Briefing

  • Could not resolve host is DNS, Could not connect or Connection timed out is the connection. Read the message first, then pick the tool.
  • Every Pod asks the cluster DNS (CoreDNS, 10.43.0.10) via /etc/resolv.conf. With ndots:5 it first tries all search domains.
  • NetworkPolicies only apply to Pods matched by their podSelector. If it matches none, nothing happens – without a warning.
  • In RKE2, Canal (Calico) enforces the policies.
Fault Symptom Where the answer is
Namespace missing from name Could not resolve host resolv.conf, search list
Default deny without DNS name fails, IP works get netpol, egress rules
Policy without effect everything allowed describe netpol, get pods -l
dnsPolicy: Default cluster names unknown .spec.dnsPolicy

Drill 1: The name without a namespace

A target api in the namespace tag08-andere (‘other’), the tool Pod client in tag08.

Break it:

# on cp1
mkdir -p ~/debug/tag08 && cd ~/debug/tag08
kubectl create namespace tag08
kubectl create namespace tag08-andere
cat > grundlage.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api, namespace: tag08-andere}
spec:
  replicas: 1
  selector: {matchLabels: {app: api}}
  template:
    metadata: {labels: {app: api}}
    spec:
      containers:
      - name: whoami
        image: traefik/whoami:v1.11
        ports: [{name: http, containerPort: 80}]
---
apiVersion: v1
kind: Service
metadata: {name: api, namespace: tag08-andere}
spec:
  selector: {app: api}
  ports: [{name: http, port: 80, targetPort: http}]
---
apiVersion: v1
kind: Pod
metadata: {name: client, namespace: tag08, labels: {app: client}}
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot:v0.15
    command: [sleep, infinity]
EOF
kubectl apply -f grundlage.yaml
kubectl -n tag08-andere rollout status deploy/api
kubectl -n tag08 wait pod/client --for=condition=Ready

Symptom:

# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api

Expected:

curl: (6) Could not resolve host: api (Domain name not found)
command terminated with exit code 6

Investigate:

# on cp1
kubectl -n tag08 exec client -- cat /etc/resolv.conf

Expected:

search tag08.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.43.0.10
options ndots:5

Cause: The short name api first becomes api.tag08.svc.cluster.local – in the Pod’s own namespace. But the Service lives in tag08-andere. The remaining search domains do not help, and localdomain comes from the node.

Fix: Across namespace boundaries, use at least <service>.<namespace>.

Verify:

# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere | head -1

Expected:

Hostname: api-c78ffd49-rwzpk

Save your progress? Sign up for free

Drill 2: dig and nslookup are not the same

# on cp1
kubectl -n tag08 exec client -- nslookup api.tag08-andere
kubectl -n tag08 exec client -- dig api.tag08-andere +noall +comments | grep status
kubectl -n tag08 exec client -- dig +search +short api.tag08-andere
kubectl -n tag08 exec client -- dig +short api.tag08-andere.svc.cluster.local

Expected (abridged):

;; Got recursion not available from 10.43.0.10
Server:     10.43.0.10
Address:    10.43.0.10#53

Name:   api.tag08-andere.svc.cluster.local
Address: 10.43.131.137
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 56523
10.43.131.137
10.43.131.137

Why? nslookup uses the search list like the application does, dig does not: it asks for api.tag08-andere. literally and gets NXDOMAIN. Only with +search or the fully qualified name (FQDN) does it behave like curl. If you test with dig without +search, you chase a fault the application does not have. The recursion not available lines are harmless: CoreDNS answers for cluster.local itself.

Save your progress? Sign up for free

Drill 3: Seeing ndots with tcpdump

What does an external name like example.com cost? tcpdump from netshoot records in the Pod while curl runs:

# on cp1
for n in example.com example.com.; do
  kubectl -n tag08 exec client -- sh -c \
    "timeout 5 tcpdump -ni eth0 -l udp port 53 >/tmp/dns.txt 2>/dev/null & sleep 1
     curl -s -o /dev/null $n; sleep 5; grep -o ' A? [^ ]*' /tmp/dns.txt"
  echo ---
done

Expected:

 A? example.com.tag08.svc.cluster.local.
 A? example.com.svc.cluster.local.
 A? example.com.cluster.local.
 A? example.com.localdomain.
 A? example.com.
---
 A? example.com.
---

Why? example.com has one dot, fewer than ndots:5, so it counts as relative: four wasted queries (A and AAAA each) before the right one. With a trailing dot the name is absolute. In production you lower ndots via dnsConfig in the Pod when many external names are resolved.

Save your progress? Sign up for free

Drill 4: Examining CoreDNS itself

In RKE2 the DNS Service is not called kube-dns but rke2-coredns-rke2-coredns. The label k8s-app=kube-dns has stayed, though:

# on cp1
kubectl -n kube-system get svc -l k8s-app=kube-dns
kubectl -n kube-system get endpointslice -l k8s-app=kube-dns
kubectl -n kube-system get pods -l k8s-app=kube-dns \
  -o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName

Expected:

NAME                        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
rke2-coredns-rke2-coredns   ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP   87m
NAME                              ADDRESSTYPE   PORTS   ENDPOINTS             AGE
rke2-coredns-rke2-coredns-p5h5d   IPv4          53,53   10.42.0.2,10.42.1.3   87m
NAME                                         IP          NODE
rke2-coredns-rke2-coredns-76dcbcb4cc-h7gdk   10.42.1.3   worker1
rke2-coredns-rke2-coredns-76dcbcb4cc-mp6sb   10.42.0.2   cp1

Out of the box, the logs only show errors (errors). Which queries arrive is revealed by the log plugin. You add it after errors in the Corefile:

Warning: The ConfigMap belongs to the RKE2 chart rke2-coredns. The change is only meant for troubleshooting and is reverted at the end of the drill. Permanent changes belong in a HelmChartConfig.

# on cp1
kubectl -n kube-system get cm rke2-coredns-rke2-coredns -o yaml > coredns-cm.yaml
sed 's/^\( *\)errors$/&\n\1log/' coredns-cm.yaml > coredns-log.yaml
diff coredns-cm.yaml coredns-log.yaml
kubectl replace -f coredns-log.yaml

Expected:

5a6
>         log
configmap/rke2-coredns-rke2-coredns replaced

The reload plugin reloads the Corefile as soon as the kubelet has mirrored the ConfigMap into the Pods – in the lab after 30 to 90 seconds. Wait until both Pods report Reloading complete, then ask for a name that does not exist (api.tag08 – the Service lives in tag08-andere):

# on cp1
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 | grep -c 'Reloading complete'
kubectl -n tag08 exec client -- curl -s -o /dev/null api.tag08
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20 | grep '"A IN api' \
  | cut -c1-88

Expected (abridged):

2
command terminated with exit code 6
[INFO] 10.42.2.30:37926 - 29717 "A IN api.tag08.tag08.svc.cluster.local. udp 74 false 1
[INFO] 10.42.2.30:37926 - 62687 "A IN api.tag08.svc.cluster.local. udp 56 false 1232" N
[INFO] 10.42.2.30:37926 - 26222 "A IN api.tag08.cluster.local. udp 52 false 1232" NXDO
[INFO] 10.42.2.30:37926 - 26786 "A IN api.tag08.localdomain. udp 50 false 1232" NXDOMA
[INFO] 10.42.2.30:37926 - 26173 "A IN api.tag08. udp 38 false 1232" NXDOMAIN qr,rd,ra

The 2 counts the reloads of both Pods; if you have changed the Corefile before, the number is higher. Now you see the search list from the server side: the query arrives, CoreDNS correctly answers NXDOMAIN. The fault lies with the name, not with CoreDNS.

Back to the original:

# on cp1
kubectl replace -f coredns-cm.yaml 2>&1 | cut -c1-60
kubectl -n kube-system get cm rke2-coredns-rke2-coredns -o yaml \
  | sed '/^ *log$/d' | kubectl replace -f -

Expected:

Error from server (Conflict): error when replacing "coredns-
configmap/rke2-coredns-rke2-coredns replaced

Why the conflict? The saved file contains the old resourceVersion. The API server rejects it because the object has changed in the meantime – protection against lost updates. So fetch the current version and remove only that line.

Save your progress? Sign up for free

Drill 5: Default deny swallows DNS

Break it: Block all outbound traffic and allow only the path to tag08-andere on port 80:

# on cp1
cat > egress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-egress, namespace: tag08}
spec:
  podSelector: {}
  policyTypes: [Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: erlaube-api, namespace: tag08}
spec:
  podSelector: {matchLabels: {app: client}}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: tag08-andere}
    ports: [{port: 80, protocol: TCP}]
EOF
kubectl apply -f egress.yaml

Symptom:

# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere
IP=$(kubectl -n tag08-andere get svc api -o jsonpath='{.spec.clusterIP}')
kubectl -n tag08 exec client -- curl -sS -m 5 $IP | head -1

Expected:

curl: (28) Resolving timed out after 5000 milliseconds
command terminated with exit code 28
Hostname: api-c78ffd49-rwzpk

The name runs into a timeout, the IP works. Not Domain name not found as in Drill 1 – no answer comes back at all.

Investigate:

# on cp1
kubectl -n tag08 exec client -- dig +short +timeout=2 +tries=1 \
  api.tag08-andere.svc.cluster.local
kubectl -n tag08 get networkpolicy
kubectl -n tag08 describe networkpolicy erlaube-api | sed -n '/Spec/,$p'

Expected:

;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached
command terminated with exit code 9
NAME                  POD-SELECTOR   AGE
default-deny-egress   <none>         12s
erlaube-api           app=client     12s
Spec:
  PodSelector:     app=client
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: 80/TCP
    To:
      NamespaceSelector: kubernetes.io/metadata.name=tag08-andere
  Policy Types: Egress

Cause: DNS is just traffic too – UDP/TCP 53 to the CoreDNS Pods in kube-system. No rule allows it, so Canal silently drops the queries.

Fix:

# on cp1
cat > dns.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: erlaube-dns, namespace: tag08}
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: kube-system}
      podSelector:
        matchLabels: {k8s-app: kube-dns}
    ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]
EOF
kubectl apply -f dns.yaml

Verify:

# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere | head -1
kubectl -n tag08 exec client -- curl -sS -m 5 -o /dev/null https://example.com

Expected:

Hostname: api-c78ffd49-rwzpk
curl: (28) Connection timed out after 5002 milliseconds
command terminated with exit code 28

The name resolves now. example.com still fails, but at the connection (Connection timed out instead of Resolving timed out) – exactly what default deny is supposed to do. Important: namespaceSelector and podSelector in the same list entry mean “AND”. As two entries (- namespaceSelector … and - podSelector …) they would mean “OR”.

Save your progress? Sign up for free

Drill 6: The policy that does nothing

Goal: only Pods from tag08 may reach api.

Break it:

# on cp1
cat > ingress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: nur-tag08, namespace: tag08-andere}
spec:
  podSelector: {matchLabels: {app: api-server}}
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: tag08}
EOF
kubectl apply -f ingress.yaml
kubectl -n tag08-andere run fremd --image=nicolaka/netshoot:v0.15 --restart=Never \
  -- sleep infinity
kubectl -n tag08-andere wait pod/fremd --for=condition=Ready

Symptom: A Pod outside tag08 still gets through:

# on cp1
kubectl -n tag08-andere exec fremd -- curl -sS -m 5 api | head -1

Expected:

Hostname: api-c78ffd49-rwzpk

Investigate:

# on cp1
kubectl -n tag08-andere describe networkpolicy nur-tag08 | grep PodSelector
kubectl -n tag08-andere get pods -l app=api-server
kubectl -n tag08-andere get pods --show-labels | cut -c1-80

Expected:

  PodSelector:     app=api-server
No resources found in tag08-andere namespace.
NAME                 READY   STATUS    RESTARTS   AGE     LABELS
api-c78ffd49-rwzpk   1/1     Running   0          4m53s   app=api,pod-template-h
fremd                1/1     Running   0          3s      run=fremd

Cause: The policy selects Pods with app=api-server – there are none. A Pod that no policy selects is open. Kubernetes does not check selectors against existing Pods; the policy is valid and has no effect.

Fix:

# on cp1
kubectl -n tag08-andere patch networkpolicy nur-tag08 --type=merge \
  -p '{"spec":{"podSelector":{"matchLabels":{"app":"api"}}}}'

Verify:

# on cp1
kubectl -n tag08-andere exec fremd -- curl -sS -m 5 api
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere | head -1

Expected:

curl: (28) Connection timed out after 5002 milliseconds
command terminated with exit code 28
Hostname: api-c78ffd49-rwzpk

Why? The opposite fault is a namespaceSelector with a label the namespace does not carry (say name: tag08): then the policy blocks everything. The only reliable label is kubernetes.io/metadata.name, which Kubernetes sets on every namespace automatically.

Talos: With the default CNI Flannel, every NetworkPolicy stays without effect, no matter how good the selectors are (Day 19).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the DNS configuration of the Pod client. Target time: 10 s
  2. Find IP and endpoints of the cluster DNS. Target time: 20 s
  3. Resolve api from tag08-andere with dig in a way that is sure to work. Target time: 20 s
  4. Show all NetworkPolicies in the cluster. Target time: 10 s

Solution

# on cp1
kubectl -n tag08 exec client -- cat /etc/resolv.conf
kubectl -n kube-system get svc -l k8s-app=kube-dns
kubectl -n kube-system get endpointslice -l k8s-app=kube-dns
kubectl -n tag08 exec client -- dig +short api.tag08-andere.svc.cluster.local
kubectl get netpol -A

Expected:

search tag08.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.43.0.10
options ndots:5
NAME                        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
rke2-coredns-rke2-coredns   ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP   92m
NAME                              ADDRESSTYPE   PORTS   ENDPOINTS             AGE
rke2-coredns-rke2-coredns-p5h5d   IPv4          53,53   10.42.0.2,10.42.1.3   92m
10.43.131.137
NAMESPACE      NAME                  POD-SELECTOR   AGE
tag08-andere   nur-tag08             app=api        87s
tag08          default-deny-egress   <none>         2m11s
tag08          erlaube-api           app=client     2m11s
tag08          erlaube-dns           <none>         109s

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere | head -1

Expected:

Hostname: api-c78ffd49-rwzpk
# on cp1
kubectl -n tag08-andere exec fremd -- curl -sS -m 3 -o /dev/null api

Expected:

curl: (28) Connection timed out after 3002 milliseconds
command terminated with exit code 28
# on cp1
kubectl -n kube-system get cm rke2-coredns-rke2-coredns \
  -o jsonpath='{.data.Corefile}' | grep -c '^ *log$'

Expected:

0

Challenge

A team has brought along an old Pod altlast (‘legacy’) in tag08-andere with dnsPolicy: Default. External names work, api in the same namespace does not. Find the cause and repair the Pod.

# on cp1
cat > altlast.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: altlast, namespace: tag08-andere}
spec:
  dnsPolicy: Default
  containers:
  - name: netshoot
    image: nicolaka/netshoot:v0.15
    command: [sleep, infinity]
EOF
kubectl apply -f altlast.yaml

Hint: Compare the resolv.conf of altlast with that of client.

Solution

# on cp1
kubectl -n tag08-andere exec altlast -- curl -sS -m 5 api
kubectl -n tag08-andere exec altlast -- curl -sS -m 5 -o /dev/null \
  -w '%{http_code}\n' https://example.com
kubectl -n tag08-andere exec altlast -- grep -v '^#' /etc/resolv.conf
kubectl -n tag08-andere get pod altlast -o jsonpath='{.spec.dnsPolicy}{"\n"}'

Expected:

curl: (6) Could not resolve host: api (Domain name not found)
command terminated with exit code 6
200
search localdomain
nameserver 172.16.0.15
nameserver 8.8.8.8
nameserver 1.1.1.1
Default

dnsPolicy: Default does not mean “the default”, it means: inherit the node’s resolv.conf. The Pod asks the lab network’s resolvers, which do not know cluster.local. The actual default is ClusterFirst. dnsPolicy is immutable, so recreate the Pod:

# on cp1
kubectl -n tag08-andere delete pod altlast
sed -i 's/dnsPolicy: Default/dnsPolicy: ClusterFirst/' altlast.yaml
kubectl apply -f altlast.yaml
kubectl -n tag08-andere wait pod/altlast --for=condition=Ready
kubectl -n tag08-andere exec altlast -- dig +search +short api
kubectl -n tag08-andere exec altlast -- curl -sS -m 5 api

Expected:

pod "altlast" deleted from tag08-andere namespace
pod/altlast created
pod/altlast condition met
10.43.131.137
curl: (28) Connection timed out after 5001 milliseconds
command terminated with exit code 28

The name resolves. That curl still fails is Drill 6 at work: altlast is not in tag08. Read first, then repair – otherwise you tear down the policy that is working exactly as intended.

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag08 tag08-andere
kubectl get netpol -A

Expected:

namespace "tag08" deleted
namespace "tag08-andere" deleted
No resources found

Interview Note

  • A name does not resolve – where do you start? resolv.conf in the Pod, then dig with the FQDN, then CoreDNS endpoints and logs (log plugin). A timeout instead of NXDOMAIN points to a NetworkPolicy or dead CoreDNS Pods.
  • What does ndots:5 do? Names with fewer than five dots go through the search list first – several queries per external name.
  • Default deny introduced, suddenly nothing works? Egress to kube-dns on 53/UDP and 53/TCP is missing.

Enterprise Note

  • Default-deny templates per namespace always include the DNS exception; new namespaces get them via GitOps or a policy engine.
  • NodeLocal DNSCache takes load off CoreDNS and makes latencies more stable; Pods then ask a local IP (169.254.20.10) instead of 10.43.0.10.
  • In large clusters the log plugin produces huge numbers of lines – switch it on only briefly and with a purpose; for the long run prefer metrics (coredns_dns_requests_total).
  • You learn to write and test policies in “Kyverno & NetworkPolicies hands-on”; here it is only about the fault pattern.