Day 8: DNS and NetworkPolicies
Goal: You tell name errors apart from connection errors, watch CoreDNS at work and find NetworkPolicies that block too much or nothing at all. · Duration: approx. 75 min · Prerequisite: Day 7
Briefing
Could not resolve hostis DNS,Could not connectorConnection timed outis the connection. Read the message first, then pick the tool.- Every Pod asks the cluster DNS (CoreDNS,
10.43.0.10) via/etc/resolv.conf. Withndots:5it first tries all search domains. - NetworkPolicies only apply to Pods matched by their
podSelector. If it matches none, nothing happens – without a warning. - In RKE2, Canal (Calico) enforces the policies.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Namespace missing from name | Could not resolve host |
resolv.conf, search list |
| Default deny without DNS | name fails, IP works | get netpol, egress rules |
| Policy without effect | everything allowed | describe netpol, get pods -l |
dnsPolicy: Default |
cluster names unknown | .spec.dnsPolicy |
Drill 1: The name without a namespace
A target api in the namespace tag08-andere
(‘other’), the tool Pod client in tag08.
Break it:
# on cp1
mkdir -p ~/debug/tag08 && cd ~/debug/tag08
kubectl create namespace tag08
kubectl create namespace tag08-andere
cat > grundlage.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api, namespace: tag08-andere}
spec:
replicas: 1
selector: {matchLabels: {app: api}}
template:
metadata: {labels: {app: api}}
spec:
containers:
- name: whoami
image: traefik/whoami:v1.11
ports: [{name: http, containerPort: 80}]
---
apiVersion: v1
kind: Service
metadata: {name: api, namespace: tag08-andere}
spec:
selector: {app: api}
ports: [{name: http, port: 80, targetPort: http}]
---
apiVersion: v1
kind: Pod
metadata: {name: client, namespace: tag08, labels: {app: client}}
spec:
containers:
- name: netshoot
image: nicolaka/netshoot:v0.15
command: [sleep, infinity]
EOF
kubectl apply -f grundlage.yaml
kubectl -n tag08-andere rollout status deploy/api
kubectl -n tag08 wait pod/client --for=condition=ReadySymptom:
# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 apiExpected:
curl: (6) Could not resolve host: api (Domain name not found)
command terminated with exit code 6
Investigate:
# on cp1
kubectl -n tag08 exec client -- cat /etc/resolv.confExpected:
search tag08.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.43.0.10
options ndots:5
Cause: The short name api first becomes
api.tag08.svc.cluster.local – in the Pod’s own namespace.
But the Service lives in tag08-andere. The remaining search
domains do not help, and localdomain comes from the
node.
Fix: Across namespace boundaries, use at least
<service>.<namespace>.
Verify:
# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere | head -1Expected:
Hostname: api-c78ffd49-rwzpk
Save your progress? Sign up for free
Drill 2: dig and nslookup are not the same
# on cp1
kubectl -n tag08 exec client -- nslookup api.tag08-andere
kubectl -n tag08 exec client -- dig api.tag08-andere +noall +comments | grep status
kubectl -n tag08 exec client -- dig +search +short api.tag08-andere
kubectl -n tag08 exec client -- dig +short api.tag08-andere.svc.cluster.localExpected (abridged):
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53
Name: api.tag08-andere.svc.cluster.local
Address: 10.43.131.137
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 56523
10.43.131.137
10.43.131.137
Why? nslookup uses the search list like the application
does, dig does not: it asks for
api.tag08-andere. literally and gets NXDOMAIN.
Only with +search or the fully qualified name (FQDN) does
it behave like curl. If you test with dig
without +search, you chase a fault the application does not
have. The recursion not available lines are harmless:
CoreDNS answers for cluster.local itself.
Save your progress? Sign up for free
Drill 3: Seeing ndots with tcpdump
What does an external name like example.com cost?
tcpdump from netshoot records in the Pod while
curl runs:
# on cp1
for n in example.com example.com.; do
kubectl -n tag08 exec client -- sh -c \
"timeout 5 tcpdump -ni eth0 -l udp port 53 >/tmp/dns.txt 2>/dev/null & sleep 1
curl -s -o /dev/null $n; sleep 5; grep -o ' A? [^ ]*' /tmp/dns.txt"
echo ---
doneExpected:
A? example.com.tag08.svc.cluster.local.
A? example.com.svc.cluster.local.
A? example.com.cluster.local.
A? example.com.localdomain.
A? example.com.
---
A? example.com.
---
Why? example.com has one dot, fewer than
ndots:5, so it counts as relative: four wasted queries (A
and AAAA each) before the right one. With a trailing dot the name is
absolute. In production you lower ndots via
dnsConfig in the Pod when many external names are
resolved.
Save your progress? Sign up for free
Drill 4: Examining CoreDNS itself
In RKE2 the DNS Service is not called kube-dns but
rke2-coredns-rke2-coredns. The label
k8s-app=kube-dns has stayed, though:
# on cp1
kubectl -n kube-system get svc -l k8s-app=kube-dns
kubectl -n kube-system get endpointslice -l k8s-app=kube-dns
kubectl -n kube-system get pods -l k8s-app=kube-dns \
-o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeNameExpected:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rke2-coredns-rke2-coredns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP 87m
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
rke2-coredns-rke2-coredns-p5h5d IPv4 53,53 10.42.0.2,10.42.1.3 87m
NAME IP NODE
rke2-coredns-rke2-coredns-76dcbcb4cc-h7gdk 10.42.1.3 worker1
rke2-coredns-rke2-coredns-76dcbcb4cc-mp6sb 10.42.0.2 cp1
Out of the box, the logs only show errors (errors).
Which queries arrive is revealed by the log plugin. You add
it after errors in the Corefile:
Warning: The ConfigMap belongs to the RKE2 chart
rke2-coredns. The change is only meant for troubleshooting and is reverted at the end of the drill. Permanent changes belong in aHelmChartConfig.
# on cp1
kubectl -n kube-system get cm rke2-coredns-rke2-coredns -o yaml > coredns-cm.yaml
sed 's/^\( *\)errors$/&\n\1log/' coredns-cm.yaml > coredns-log.yaml
diff coredns-cm.yaml coredns-log.yaml
kubectl replace -f coredns-log.yamlExpected:
5a6
> log
configmap/rke2-coredns-rke2-coredns replaced
The reload plugin reloads the Corefile as soon as the
kubelet has mirrored the ConfigMap into the Pods – in the lab after 30
to 90 seconds. Wait until both Pods report
Reloading complete, then ask for a name that does not exist
(api.tag08 – the Service lives in
tag08-andere):
# on cp1
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 | grep -c 'Reloading complete'
kubectl -n tag08 exec client -- curl -s -o /dev/null api.tag08
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20 | grep '"A IN api' \
| cut -c1-88Expected (abridged):
2
command terminated with exit code 6
[INFO] 10.42.2.30:37926 - 29717 "A IN api.tag08.tag08.svc.cluster.local. udp 74 false 1
[INFO] 10.42.2.30:37926 - 62687 "A IN api.tag08.svc.cluster.local. udp 56 false 1232" N
[INFO] 10.42.2.30:37926 - 26222 "A IN api.tag08.cluster.local. udp 52 false 1232" NXDO
[INFO] 10.42.2.30:37926 - 26786 "A IN api.tag08.localdomain. udp 50 false 1232" NXDOMA
[INFO] 10.42.2.30:37926 - 26173 "A IN api.tag08. udp 38 false 1232" NXDOMAIN qr,rd,ra
The 2 counts the reloads of both Pods; if you have
changed the Corefile before, the number is higher. Now you see the
search list from the server side: the query arrives, CoreDNS correctly
answers NXDOMAIN. The fault lies with the name, not with
CoreDNS.
Back to the original:
# on cp1
kubectl replace -f coredns-cm.yaml 2>&1 | cut -c1-60
kubectl -n kube-system get cm rke2-coredns-rke2-coredns -o yaml \
| sed '/^ *log$/d' | kubectl replace -f -Expected:
Error from server (Conflict): error when replacing "coredns-
configmap/rke2-coredns-rke2-coredns replaced
Why the conflict? The saved file contains the old
resourceVersion. The API server rejects it because the
object has changed in the meantime – protection against lost updates. So
fetch the current version and remove only that line.
Save your progress? Sign up for free
Drill 5: Default deny swallows DNS
Break it: Block all outbound traffic and allow only
the path to tag08-andere on port 80:
# on cp1
cat > egress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-egress, namespace: tag08}
spec:
podSelector: {}
policyTypes: [Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: erlaube-api, namespace: tag08}
spec:
podSelector: {matchLabels: {app: client}}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag08-andere}
ports: [{port: 80, protocol: TCP}]
EOF
kubectl apply -f egress.yamlSymptom:
# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere
IP=$(kubectl -n tag08-andere get svc api -o jsonpath='{.spec.clusterIP}')
kubectl -n tag08 exec client -- curl -sS -m 5 $IP | head -1Expected:
curl: (28) Resolving timed out after 5000 milliseconds
command terminated with exit code 28
Hostname: api-c78ffd49-rwzpk
The name runs into a timeout, the IP works. Not
Domain name not found as in Drill 1 – no answer comes back
at all.
Investigate:
# on cp1
kubectl -n tag08 exec client -- dig +short +timeout=2 +tries=1 \
api.tag08-andere.svc.cluster.local
kubectl -n tag08 get networkpolicy
kubectl -n tag08 describe networkpolicy erlaube-api | sed -n '/Spec/,$p'Expected:
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached
command terminated with exit code 9
NAME POD-SELECTOR AGE
default-deny-egress <none> 12s
erlaube-api app=client 12s
Spec:
PodSelector: app=client
Not affecting ingress traffic
Allowing egress traffic:
To Port: 80/TCP
To:
NamespaceSelector: kubernetes.io/metadata.name=tag08-andere
Policy Types: Egress
Cause: DNS is just traffic too – UDP/TCP 53 to the
CoreDNS Pods in kube-system. No rule allows it, so Canal
silently drops the queries.
Fix:
# on cp1
cat > dns.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: erlaube-dns, namespace: tag08}
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
podSelector:
matchLabels: {k8s-app: kube-dns}
ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]
EOF
kubectl apply -f dns.yamlVerify:
# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere | head -1
kubectl -n tag08 exec client -- curl -sS -m 5 -o /dev/null https://example.comExpected:
Hostname: api-c78ffd49-rwzpk
curl: (28) Connection timed out after 5002 milliseconds
command terminated with exit code 28
The name resolves now. example.com still fails, but at
the connection (Connection timed out
instead of Resolving timed out) – exactly what default deny
is supposed to do. Important: namespaceSelector and
podSelector in the same list entry mean “AND”. As two
entries (- namespaceSelector … and
- podSelector …) they would mean “OR”.
Save your progress? Sign up for free
Drill 6: The policy that does nothing
Goal: only Pods from tag08 may reach
api.
Break it:
# on cp1
cat > ingress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: nur-tag08, namespace: tag08-andere}
spec:
podSelector: {matchLabels: {app: api-server}}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag08}
EOF
kubectl apply -f ingress.yaml
kubectl -n tag08-andere run fremd --image=nicolaka/netshoot:v0.15 --restart=Never \
-- sleep infinity
kubectl -n tag08-andere wait pod/fremd --for=condition=ReadySymptom: A Pod outside tag08 still gets
through:
# on cp1
kubectl -n tag08-andere exec fremd -- curl -sS -m 5 api | head -1Expected:
Hostname: api-c78ffd49-rwzpk
Investigate:
# on cp1
kubectl -n tag08-andere describe networkpolicy nur-tag08 | grep PodSelector
kubectl -n tag08-andere get pods -l app=api-server
kubectl -n tag08-andere get pods --show-labels | cut -c1-80Expected:
PodSelector: app=api-server
No resources found in tag08-andere namespace.
NAME READY STATUS RESTARTS AGE LABELS
api-c78ffd49-rwzpk 1/1 Running 0 4m53s app=api,pod-template-h
fremd 1/1 Running 0 3s run=fremd
Cause: The policy selects Pods with
app=api-server – there are none. A Pod that no policy
selects is open. Kubernetes does not check selectors against existing
Pods; the policy is valid and has no effect.
Fix:
# on cp1
kubectl -n tag08-andere patch networkpolicy nur-tag08 --type=merge \
-p '{"spec":{"podSelector":{"matchLabels":{"app":"api"}}}}'Verify:
# on cp1
kubectl -n tag08-andere exec fremd -- curl -sS -m 5 api
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere | head -1Expected:
curl: (28) Connection timed out after 5002 milliseconds
command terminated with exit code 28
Hostname: api-c78ffd49-rwzpk
Why? The opposite fault is a namespaceSelector with a
label the namespace does not carry (say name: tag08): then
the policy blocks everything. The only reliable label
is kubernetes.io/metadata.name, which Kubernetes sets on
every namespace automatically.
Talos: With the default CNI Flannel, every NetworkPolicy stays without effect, no matter how good the selectors are (Day 19).
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the DNS configuration of the Pod
client. Target time: 10 s - Find IP and endpoints of the cluster DNS. Target time: 20 s
- Resolve
apifromtag08-anderewithdigin a way that is sure to work. Target time: 20 s - Show all NetworkPolicies in the cluster. Target time: 10 s
Solution
# on cp1
kubectl -n tag08 exec client -- cat /etc/resolv.conf
kubectl -n kube-system get svc -l k8s-app=kube-dns
kubectl -n kube-system get endpointslice -l k8s-app=kube-dns
kubectl -n tag08 exec client -- dig +short api.tag08-andere.svc.cluster.local
kubectl get netpol -AExpected:
search tag08.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.43.0.10
options ndots:5
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rke2-coredns-rke2-coredns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP 92m
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
rke2-coredns-rke2-coredns-p5h5d IPv4 53,53 10.42.0.2,10.42.1.3 92m
10.43.131.137
NAMESPACE NAME POD-SELECTOR AGE
tag08-andere nur-tag08 app=api 87s
tag08 default-deny-egress <none> 2m11s
tag08 erlaube-api app=client 2m11s
tag08 erlaube-dns <none> 109s
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag08 exec client -- curl -sS -m 5 api.tag08-andere | head -1Expected:
Hostname: api-c78ffd49-rwzpk
# on cp1
kubectl -n tag08-andere exec fremd -- curl -sS -m 3 -o /dev/null apiExpected:
curl: (28) Connection timed out after 3002 milliseconds
command terminated with exit code 28
# on cp1
kubectl -n kube-system get cm rke2-coredns-rke2-coredns \
-o jsonpath='{.data.Corefile}' | grep -c '^ *log$'Expected:
0
Challenge
A team has brought along an old Pod altlast (‘legacy’)
in tag08-andere with dnsPolicy: Default.
External names work, api in the same namespace does not.
Find the cause and repair the Pod.
# on cp1
cat > altlast.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: altlast, namespace: tag08-andere}
spec:
dnsPolicy: Default
containers:
- name: netshoot
image: nicolaka/netshoot:v0.15
command: [sleep, infinity]
EOF
kubectl apply -f altlast.yamlHint: Compare the resolv.conf of
altlast with that of client.
Solution
# on cp1
kubectl -n tag08-andere exec altlast -- curl -sS -m 5 api
kubectl -n tag08-andere exec altlast -- curl -sS -m 5 -o /dev/null \
-w '%{http_code}\n' https://example.com
kubectl -n tag08-andere exec altlast -- grep -v '^#' /etc/resolv.conf
kubectl -n tag08-andere get pod altlast -o jsonpath='{.spec.dnsPolicy}{"\n"}'Expected:
curl: (6) Could not resolve host: api (Domain name not found)
command terminated with exit code 6
200
search localdomain
nameserver 172.16.0.15
nameserver 8.8.8.8
nameserver 1.1.1.1
Default
dnsPolicy: Default does not mean “the default”, it
means: inherit the node’s resolv.conf. The Pod asks the lab
network’s resolvers, which do not know cluster.local. The
actual default is ClusterFirst. dnsPolicy is
immutable, so recreate the Pod:
# on cp1
kubectl -n tag08-andere delete pod altlast
sed -i 's/dnsPolicy: Default/dnsPolicy: ClusterFirst/' altlast.yaml
kubectl apply -f altlast.yaml
kubectl -n tag08-andere wait pod/altlast --for=condition=Ready
kubectl -n tag08-andere exec altlast -- dig +search +short api
kubectl -n tag08-andere exec altlast -- curl -sS -m 5 apiExpected:
pod "altlast" deleted from tag08-andere namespace
pod/altlast created
pod/altlast condition met
10.43.131.137
curl: (28) Connection timed out after 5001 milliseconds
command terminated with exit code 28
The name resolves. That curl still fails is Drill 6 at
work: altlast is not in tag08. Read first,
then repair – otherwise you tear down the policy that is working exactly
as intended.
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag08 tag08-andere
kubectl get netpol -AExpected:
namespace "tag08" deleted
namespace "tag08-andere" deleted
No resources found
Interview Note
- A name does not resolve – where do you start?
resolv.confin the Pod, thendigwith the FQDN, then CoreDNS endpoints and logs (logplugin). A timeout instead ofNXDOMAINpoints to a NetworkPolicy or dead CoreDNS Pods. - What does
ndots:5do? Names with fewer than five dots go through the search list first – several queries per external name. - Default deny introduced, suddenly nothing works? Egress to
kube-dnson 53/UDP and 53/TCP is missing.
Enterprise Note
- Default-deny templates per namespace always include the DNS exception; new namespaces get them via GitOps or a policy engine.
- NodeLocal DNSCache takes load off CoreDNS and makes latencies more
stable; Pods then ask a local IP (
169.254.20.10) instead of10.43.0.10. - In large clusters the
logplugin produces huge numbers of lines – switch it on only briefly and with a purpose; for the long run prefer metrics (coredns_dns_requests_total). - You learn to write and test policies in “Kyverno & NetworkPolicies hands-on”; here it is only about the fault pattern.