Day 8: Customizing CoreDNS
Goal: You read and change the Corefile safely, understand Service and Pod names, ndots and dnsConfig, and find DNS errors quickly. · Duration: approx. 90 min · Prerequisite: Day 6
Briefing
- Curriculum: “CKNE: Service Networking and DNS” (Customizing CoreDNS) and “Troubleshooting Pod Connectivity” (DNS).
- CoreDNS is a perfectly ordinary Deployment behind the Service
kube-dns(10.96.0.10); its configuration lives in the ConfigMapcoredns. - The kubelet writes
nameserver 10.96.0.10, the search list andndots:5into every Pod’s/etc/resolv.conf. - The
reloadplugin picks up Corefile changes without restarting Pods – a typo can still take down cluster DNS. That’s why today’s rule is: check first, then apply.
Pod (resolv.conf: 10.96.0.10, ndots:5)
| Cilium Socket-LB: 10.96.0.10:53 -> CoreDNS Pod
v
CoreDNS (ConfigMap coredns, Corefile)
|-- cluster.local -> kubernetes plugin (Services, Pods)
|-- lab.intern -> forward 172.16.0.123 (router, Drill 6)
'-- everything else -> forward the node's /etc/resolv.conf
Drill 1: Read the CoreDNS setup
First you back up the original ConfigMap – at the end of the day you restore exactly this.
mkdir -p ~/ckne/tag08 && cd ~/ckne/tag08
k -n kube-system get cm coredns -o yaml > ~/ckne/tag08/coredns-vorher.yaml
k -n kube-system get deploy,svc,endpointslices -l k8s-app=kube-dnsExpected (abridged):
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 2/2 2 2 41m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 41m
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kube-dns-jjx5f IPv4 53,53,9153 10.0.0.219,10.0.0.71 41m
For historical reasons the Service is called kube-dns;
CoreDNS runs behind it. How does a Pod know the IP? From the kubelet
configuration:
sudo grep -A1 -E 'clusterDNS|clusterDomain|resolvConf' /var/lib/kubelet/config.yaml
grep -v '^#' /run/systemd/resolve/resolv.confExpected (abridged):
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
resolvConf: /run/systemd/resolve/resolv.conf
nameserver 172.16.0.15
nameserver 8.8.8.8
nameserver 1.1.1.1
search localdomain
resolvConf is the file that Pods with
dnsPolicy: Default get – including CoreDNS itself
(k -n kube-system get deploy coredns -o jsonpath='{..dnsPolicy}'
→ Default). That’s where the upstream servers for external
names come from. Now the Corefile:
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}'Expected:
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30 {
disable success cluster.local
disable denial cluster.local
}
loop
reload
loadbalance
}
| Plugin | Purpose |
|---|---|
errors, log |
errors or every query, respectively, to the container log |
health, ready |
liveness (:8080) and readiness (:8181) of the Pod |
kubernetes |
answers cluster.local and reverse zones from the
API |
forward |
everything else to upstream servers |
cache, loop, reload,
loadbalance |
cache, loop protection, reloading, rotation |
.:53 is a server block for the root zone, so for all
names. The order of plugins in the block doesn’t matter – CoreDNS
processes them in a fixed order compiled into the binary. Finally, a
test ground:
k create namespace tag08
k -n tag08 create deployment web --image=nginx:1.29 --replicas=2
k -n tag08 expose deployment web --port=80
k -n tag08 expose deployment web --name=web-hl --cluster-ip=None --port=80
k -n tag08 run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k -n tag08 wait --for=condition=Ready pod --all --timeout=120s
k -n tag08 exec client -- cat /etc/resolv.confExpected:
search tag08.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5
Save your progress? Sign up for free
Drill 2: Service and Pod names
k -n tag08 get pods,svc -o wide
k -n tag08 exec client -- dig +short web.tag08.svc.cluster.local
k -n tag08 exec client -- dig +short web-hl.tag08.svc.cluster.local
k -n tag08 exec client -- dig +short web
k -n tag08 exec client -- dig +short +search webExpected (abridged):
pod/client 1/1 Running 0 48s 10.0.2.84 worker2
pod/web-689b9bdbdd-c2t4c 1/1 Running 0 49s 10.0.1.121 worker1
pod/web-689b9bdbdd-ch5rc 1/1 Running 0 49s 10.0.2.231 worker2
service/web ClusterIP 10.109.153.18 <none> 80/TCP 49s app=web
service/web-hl ClusterIP None <none> 80/TCP 0s app=web
10.109.153.18
10.0.2.231
10.0.1.121
10.109.153.18
Normal Service → one A answer with the ClusterIP; headless → the Pod
IPs. dig web stays empty: dig ignores the search
list; only with +search (or in
nslookup, curl, applications) does
web become web.tag08.svc.cluster.local.
SRV records exist only for named ports.
expose doesn’t assign a name, so add one:
k -n tag08 patch svc web --type=json \
-p '[{"op":"add","path":"/spec/ports/0/name","value":"http"}]'
k -n tag08 exec client -- dig +short SRV _http._tcp.web.tag08.svc.cluster.local
k -n tag08 exec client -- dig +short 10-0-1-121.tag08.pod.cluster.local
k -n tag08 exec client -- dig +short -x 10.109.153.18
k -n tag08 exec client -- dig +short -x 10.0.1.121Expected:
0 100 80 web.tag08.svc.cluster.local.
10.0.1.121
web.tag08.svc.cluster.local.
10-0-1-121.web.tag08.svc.cluster.local.
10-0-1-121.web-hl.tag08.svc.cluster.local.
Because of pods insecure, CoreDNS returns Pod A records
(<ip-with-dashes>.<ns>.pod.cluster.local) for
any IP, without checking whether the Pod exists. You get stable Pod
names via hostname and subdomain plus a
headless Service:
k -n tag08 apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: db
labels:
app: web
spec:
hostname: db
subdomain: web-hl
containers:
- name: nginx
image: nginx:1.29
EOF
k -n tag08 wait --for=condition=Ready pod/db --timeout=60s
k -n tag08 exec client -- dig +short db.web-hl.tag08.svc.cluster.local
k -n tag08 exec db -- hostname -fExpected:
10.0.1.198
db.web-hl.tag08.svc.cluster.local
This is also how StatefulSet Pods get their names
(web-0.<service>…): subdomain must match
the name of a headless Service that selects the Pod.
Save your progress? Sign up for free
Drill 3: Watch the log plugin and reload
Before you change the Corefile, you fetch CoreDNS as a binary onto
cp1 – in the same version as the image. That lets you check
the syntax before the cluster sees it. First read the version from the
Deployment:
k -n kube-system get deploy coredns -o jsonpath='{..image}'; echoExpected:
registry.k8s.io/coredns/coredns:v1.14.6 – hence
V=1.14.6:
cd /tmp
V=1.14.6
ARCH=$(dpkg --print-architecture)
U=https://github.com/coredns/coredns/releases/download/v$V
curl -sSLO $U/coredns_${V}_linux_${ARCH}.tgz
curl -sSLO $U/coredns_${V}_linux_${ARCH}.tgz.sha256
sha256sum -c coredns_${V}_linux_${ARCH}.tgz.sha256
sudo tar xzf coredns_${V}_linux_${ARCH}.tgz --no-same-owner -C /usr/local/bin
coredns -version
cd ~/ckne/tag08Expected:
coredns_1.14.6_linux_amd64.tgz: OK
CoreDNS-1.14.6
linux/amd64, go1.26.5, 424d125
Fetch the Corefile, insert log after
errors, check it:
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
sed -i 's/^ errors$/ errors\n log/' Corefile
head -3 Corefile
coredns -conf CorefileExpected:
.:53 {
errors
log
maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
plugin/kubernetes: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and
KUBERNETES_SERVICE_PORT must be defined
Here this message is the good result: the Corefile
is syntactically fine, only the kubernetes plugin finds no API outside
the cluster. A typo would look like this:
Corefile:13 - Error during parsing: Unknown directive 'forwrd'.
Limitation: errors in plugins that CoreDNS sets up only after
kubernetes (such as arguments of forward)
don’t show up here.
Apply it and watch the reload:
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -
date +%T
k -n kube-system logs -l k8s-app=kube-dns -f --tail=0 --prefixExpected (abridged, after approx. 1 min, stop with Ctrl+C):
configmap/coredns replaced
23:19:45
[pod/coredns-559f6c778d-nz7t6/coredns] [INFO] Reloading
[pod/coredns-559f6c778d-nz7t6/coredns] [INFO] plugin/reload: Running configuration
SHA512 = 2dd49c56b94cd567...
[pod/coredns-559f6c778d-nz7t6/coredns] [INFO] Reloading complete
[pod/coredns-559f6c778d-nz7t6/coredns] [INFO] 127.0.0.1:43758 - 562 "HINFO IN
4160184936247868763.2761932811795487505. udp 57 false 512" NXDOMAIN ...
Why does this take up to a minute? The kubelet updates mounted
ConfigMaps with a delay, and by default reload checks the
file every 30 s (with some jitter). Each Pod reloads on its
own – for a short while the two replicas answer differently.
The HINFO query comes from the loop plugin, which tests for
loops after every load. The kubectl edit cm coredns variant
works the same way, just without a syntax check first.
Save your progress? Sign up for free
Drill 4: ndots, dnsPolicy and dnsConfig
With log you can now see what a harmless
curl http://example.com costs:
CLIENT_IP=$(k -n tag08 get pod client -o jsonpath='{.status.podIP}')
k -n tag08 exec client -- curl -s -o /dev/null -w '%{http_code}\n' http://example.com
k -n kube-system logs -l k8s-app=kube-dns --since=20s --tail=-1 \
| grep "$CLIENT_IP" | grep '"A IN'Expected (abridged):
200
[INFO] 10.0.2.84:53716 - 59923 "A IN example.com.tag08.svc.cluster.local. ..." NXDOMAIN
[INFO] 10.0.2.84:53716 - 33823 "A IN example.com.svc.cluster.local. ..." NXDOMAIN
[INFO] 10.0.2.84:53716 - 64596 "A IN example.com.cluster.local. ..." NXDOMAIN
[INFO] 10.0.2.84:53716 - 4669 "A IN example.com.localdomain. ..." NXDOMAIN
[INFO] 10.0.2.84:53716 - 2726 "A IN example.com. udp 40 false 1232" NOERROR ...
example.com has fewer than 5 dots
(ndots:5), so the resolver first tries all four search
domains – with AAAA that’s 10 queries instead of 2.
--tail=-1 is needed because otherwise logs -l
shows only 10 lines per Pod. The per-Pod remedy:
dnsConfig.
cat > client-ndots.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: client-ndots
spec:
dnsConfig:
options:
- name: ndots
value: "1"
containers:
- name: netshoot
image: nicolaka/netshoot:v0.16
command: ["sleep", "infinity"]
EOF
cat > client-none.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: client-none
spec:
dnsPolicy: None
dnsConfig:
nameservers: ["1.1.1.1"]
searches: ["example.org"]
containers:
- name: netshoot
image: nicolaka/netshoot:v0.16
command: ["sleep", "infinity"]
EOF
k -n tag08 apply -f client-ndots.yaml -f client-none.yaml
k -n tag08 wait --for=condition=Ready pod/client-ndots pod/client-none
k -n tag08 exec client-ndots -- cat /etc/resolv.conf
k -n tag08 exec client-none -- cat /etc/resolv.confExpected:
search tag08.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:1
search example.org
nameserver 1.1.1.1
dnsConfig is merged with the
dnsPolicy; only with None does
dnsConfig alone count. The effects in the log:
NIP=$(k -n tag08 get pod client-ndots -o jsonpath='{.status.podIP}')
k -n tag08 exec client-ndots -- curl -s -o /dev/null -w '%{http_code}\n' \
http://example.com
k -n tag08 exec client-ndots -- curl -s -o /dev/null -w '%{http_code}\n' \
http://web.tag08
k -n kube-system logs -l k8s-app=kube-dns --since=20s --tail=-1 \
| grep "$NIP" | grep '"A IN'
k -n tag08 exec client-none -- curl -s -m 5 http://web.tag08.svc.cluster.local
echo $?Expected (abridged):
200
200
[INFO] 10.0.1.178:58941 - 17739 "A IN example.com. ..." NOERROR qr,rd,ra 83 0.0104s
[INFO] 10.0.1.178:52099 - 48594 "A IN web.tag08. ..." NXDOMAIN qr,rd,ra 102 0.4078s
[INFO] 10.0.1.178:52099 - 7446 "A IN web.tag08.tag08.svc.cluster.local. ..." NXDOMAIN
[INFO] 10.0.1.178:52099 - 51482 "A IN web.tag08.svc.cluster.local. ..." NOERROR
command terminated with exit code 6
6
With ndots:1, example.com goes out
immediately. The price: web.tag08 now also has a dot, so it
is first queried on the internet as an absolute name (0.4 s until
NXDOMAIN) and only then found via the search list.
client-none doesn’t know cluster DNS at all (curl exit 6 =
name could not be resolved). Without changing the Pod, a trailing dot
helps: curl http://example.com. produces exactly one A
query.
Save your progress? Sign up for free
Drill 5: hosts and rewrite
Two typical exam requests: a fixed name for an IP outside the cluster
and an alias for a Service. Open the Corefile
(vim Corefile) and insert below ready:
hosts {
172.16.0.123 router.lab.local
fallthrough
}
rewrite name exact notes.lab.local web.tag08.svc.cluster.local
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | diff - Corefile
coredns -conf Corefile 2>&1 | tail -1
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -Expected (abridged):
7a8,12
> hosts {
> 172.16.0.123 router.lab.local
> fallthrough
> }
> rewrite name exact notes.lab.local web.tag08.svc.cluster.local
plugin/kubernetes: unable to load in-cluster configuration, ...
configmap/coredns replaced
After about a minute (log: Reloading complete for both
Pods):
k -n tag08 exec client -- dig +short router.lab.local
k -n tag08 exec client -- dig +short -x 172.16.0.123
k -n tag08 exec client -- dig notes.lab.local | grep -A1 'ANSWER SECTION'
k -n tag08 exec client -- curl -s -o /dev/null -w '%{http_code}\n' \
http://notes.lab.local
k -n tag08 exec client -- dig +short example.comExpected:
172.16.0.123
router.lab.local.
;; ANSWER SECTION:
notes.lab.local. 30 IN A 10.109.153.18
200
172.66.147.243
104.20.23.154
hosts also creates the PTR record. rewrite
rewrites the question before the kubernetes plugin sees it, and rewrites
the answer back – the client keeps seeing notes.lab.local.
Without fallthrough, hosts
answers every other name with SERVFAIL (you test that in the challenge,
not on the cluster DNS).
Save your progress? Sign up for free
Drill 6: Stub domain to the router
Corporate networks have their own zones that only an internal DNS
server knows. On the router you play that server – with the
same CoreDNS binary:
ssh [email protected]
cd /tmp
V=1.14.6
ARCH=$(dpkg --print-architecture)
U=https://github.com/coredns/coredns/releases/download/v$V
curl -sSLO $U/coredns_${V}_linux_${ARCH}.tgz
curl -sSLO $U/coredns_${V}_linux_${ARCH}.tgz.sha256
sha256sum -c coredns_${V}_linux_${ARCH}.tgz.sha256
sudo tar xzf coredns_${V}_linux_${ARCH}.tgz --no-same-owner -C /usr/local/bin
sudo mkdir -p /etc/coredns
cat <<'EOF' | sudo tee /etc/coredns/Corefile
lab.intern:53 {
bind 172.16.0.123
hosts {
172.16.0.123 router.lab.intern
10.20.30.40 db.lab.intern
}
log
errors
}
EOF
sudo systemd-run --unit=tag08-dns /usr/local/bin/coredns -conf /etc/coredns/Corefile
sudo ss -lunp | grep coredns
exitExpected (abridged):
coredns_1.14.6_linux_amd64.tgz: OK
Running as unit: tag08-dns.service; invocation ID: 4ea3e335ed444ec5a3759c0995aea187
UNCONN 0 0 172.16.0.123:53 0.0.0.0:* users:(("coredns",...))
bind 172.16.0.123 is needed because
systemd-resolved already listens on 127.0.0.53:53.
systemd-run starts a transient unit that you can stop later
with one command. Back on cp1: before and after.
dig +short @172.16.0.123 db.lab.intern
k -n tag08 exec client -- dig db.lab.intern | grep status
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
cat >> Corefile <<'EOF'
lab.intern:53 {
errors
cache 30
forward . 172.16.0.123
}
EOF
coredns -conf Corefile 2>&1 | tail -1
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -Expected (abridged):
10.20.30.40
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36252
plugin/kubernetes: unable to load in-cluster configuration, ...
configmap/coredns replaced
After the reload:
k -n tag08 exec client -- dig db.lab.intern | grep -A1 -E 'status|ANSWER SEC'
k -n tag08 exec client -- dig +short router.lab.intern
ssh [email protected] "sudo journalctl -u tag08-dns --no-pager | tail -2"Expected (abridged):
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6996
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; ANSWER SECTION:
db.lab.intern. 30 IN A 10.20.30.40
172.16.0.123
... coredns[3576]: [INFO] 172.16.0.126:50139 - 35812 "A IN db.lab.intern. ..." NOERROR
... coredns[3576]: [INFO] 172.16.0.126:50139 - 9092 "A IN router.lab.intern. ..." ...
The longest matching server block wins: lab.intern goes
to the router, everything else continues through .:53. The
router sees the node IP (here cp1) as the source, because
Cilium masquerades Pod traffic leaving the cluster. This is the Corefile
form of the “stub domain”; the same line
forward . <IP> in the .:53 block would
redirect all external names (“upstream
nameserver”).
Save your progress? Sign up for free
Drill 7: Scaling CoreDNS, anti-affinity
k -n kube-system get pods -l k8s-app=kube-dns -o wide
k -n kube-system get deploy coredns -o jsonpath='{.spec.template.spec.affinity}'; echoExpected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
coredns-559f6c778d-nz7t6 1/1 Running 0 48m 10.0.0.71 cp1
coredns-559f6c778d-vwrtf 1/1 Running 0 48m 10.0.0.219 cp1
{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTer
m":{"labelSelector":...,"topologyKey":"kubernetes.io/hostname"},"weight":100}]}}
Both replicas on cp1: they were created before the
workers joined, and the anti-affinity is only preferred. If
cp1 fails, DNS is gone. Scaling redistributes:
k -n kube-system scale deployment coredns --replicas=3
k -n kube-system rollout status deployment coredns
k -n kube-system get pods -l k8s-app=kube-dns -o wide
k -n kube-system scale deployment coredns --replicas=2
k -n kube-system get pods -l k8s-app=kube-dns -o wideExpected (abridged):
coredns-559f6c778d-6png2 1/1 Running 0 6s 10.0.1.49 worker1
coredns-559f6c778d-nz7t6 1/1 Running 0 48m 10.0.0.71 cp1
coredns-559f6c778d-vwrtf 1/1 Running 0 48m 10.0.0.219 cp1
deployment.apps/coredns scaled
coredns-559f6c778d-6png2 1/1 Running 0 15s 10.0.1.49 worker1
coredns-559f6c778d-vwrtf 1/1 Running 0 48m 10.0.0.219 cp1
When scaling down, the ReplicaSet controller prefers deleting Pods on
nodes with several replicas – now they are spread out. The new Pod reads
the current Corefile; lameduck 5s lets the deleted Pod
still answer in-flight queries. For strict distribution:
requiredDuringScheduling… or
topologySpreadConstraints, plus a PodDisruptionBudget.
Save your progress? Sign up for free
Drill 8: NodeLocal DNSCache and Cilium
NodeLocal DNSCache (kubernetes.io: “Using NodeLocal DNSCache in Kubernetes Clusters”) is a DaemonSet with a CoreDNS cache on every node. Classically it listens on a link-local IP (169.254.20.10) and uses iptables to also intercept packets to the kube-dns IP. Advantages: lower latency, no conntrack problems with UDP, less load on CoreDNS.
With Cilium KPR this trick doesn’t work: the Socket-LB translates
10.96.0.10 into a CoreDNS Pod IP already at connect() in
the Pod, so the iptables rules never see the Service IP. That’s why
docs.cilium.io describes its own approach under “Local Redirect Policy”
→ “Node-local DNS cache”: node-local-dns as a normal Pod
(hostNetwork: false, -setupiptables=false)
plus a CiliumLocalRedirectPolicy that redirects the
Service kube-dns to the cache Pod on the same
node. It requires the Helm value
localRedirectPolicies.enabled=true – disabled in this
lab:
cilium config view | grep -ci redirect
k get crd | grep -c ciliumlocalredirectpoliciesExpected:
0
0
The policy from the Cilium example
(examples/kubernetes-local-redirect/) looks like this –
read only, don’t apply:
apiVersion: cilium.io/v2
kind: CiliumLocalRedirectPolicy
metadata:
name: nodelocaldns
namespace: kube-system
spec:
redirectFrontend:
serviceMatcher:
serviceName: kube-dns
namespace: kube-system
redirectBackend:
localEndpointSelector:
matchLabels:
k8s-app: node-local-dns
toPorts:
- port: "53"
name: dns
protocol: UDP
- port: "53"
name: dns-tcp
protocol: TCPAfterwards, cilium-dbg service list shows the type
LocalRedirect for 10.96.0.10 with only local backends. If
the cache Pod is missing, Cilium falls back to the normal CoreDNS
backends.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the cluster DNS IP and cluster domain that the kubelet on
cp1gives to Pods. Target time: 20 s - In
tag08, start the Podfqdn(netshoot) withndots:2. Target time: 2 min - Resolve the Pod DNS name of a web Pod. Target time: 1 min
- Show which upstream servers CoreDNS forwards external names to. Target time: 1 min
- Check from Cilium’s point of view whether
kube-dnshas backends. Target time: 30 s
Solution
sudo grep -A1 -E 'clusterDNS|clusterDomain' /var/lib/kubelet/config.yaml
k -n tag08 run fqdn --image=nicolaka/netshoot:v0.16 $do -- sleep infinity > fqdn.yaml
vim fqdn.yamlAdd below dnsPolicy: ClusterFirst:
dnsConfig:
options:
- name: ndots
value: "2"k apply -f fqdn.yaml
k -n tag08 exec fqdn -- grep options /etc/resolv.conf
P=$(k -n tag08 get pod -l app=web,pod-template-hash \
-o jsonpath='{.items[0].status.podIP}')
k -n tag08 exec client -- dig +short ${P//./-}.tag08.pod.cluster.local
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep forward
sudo grep resolvConf /var/lib/kubelet/config.yaml
grep ^nameserver /run/systemd/resolve/resolv.conf
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A2 '10.96.0.10:53/UDP'Expected (abridged):
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
options ndots:2
10.0.1.121
forward . /etc/resolv.conf {
resolvConf: /run/systemd/resolve/resolv.conf
nameserver 172.16.0.15
nameserver 8.8.8.8
nameserver 1.1.1.1
5 10.96.0.10:53/UDP ClusterIP 1 => 10.0.0.219:53/UDP (active)
2 => 10.0.1.49:53/UDP (active)
/etc/resolv.conf in the forward entry is
the file inside the CoreDNS Pod – because of
dnsPolicy: Default, the node’s resolvConf
file. For network problems, ask a CoreDNS Pod directly
(dig @10.0.0.219 …) to separate the Service path from
CoreDNS.
Save your progress? Sign up for free
Checkpoint
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' \
| grep -E 'log|hosts|rewrite|lab.intern'Expected: the lines log,
hosts {, rewrite name exact … and
lab.intern:53 {.
k -n tag08 exec client -- dig +short router.lab.localExpected: 172.16.0.123
k -n tag08 exec client -- dig +short notes.lab.localExpected: the ClusterIP of web (here
10.109.153.18).
k -n tag08 exec client -- dig +short db.lab.internExpected: 10.20.30.40
k -n tag08 exec client-ndots -- grep options /etc/resolv.confExpected: options ndots:1
k -n kube-system get pods -l k8s-app=kube-dns -o wideExpected: two CoreDNS Pods Running on
different nodes.
Challenge
A copy of CoreDNS (dns-test in tag08) no
longer starts after a restart, and even once repaired it doesn’t answer
external names. Get it running without touching the cluster DNS:
test.lab.local → 10.20.30.40,
example.com resolvable. Preparation (first the working
copy, then the broken configuration):
cat > dns-test.Corefile <<'EOF'
.:53 {
errors
log
health {
lameduck 5s
}
ready
hosts {
10.20.30.40 test.lab.local
fallthrough
}
forward . /etc/resolv.conf
cache 30
reload
}
EOF
k -n tag08 create configmap dns-test --from-file=Corefile=dns-test.Corefile
cat > dns-test.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: dns-test
spec:
replicas: 1
selector:
matchLabels:
app: dns-test
template:
metadata:
labels:
app: dns-test
spec:
containers:
- name: coredns
image: registry.k8s.io/coredns/coredns:v1.14.6
args: ["-conf", "/etc/coredns/Corefile"]
readinessProbe:
httpGet:
path: /ready
port: 8181
volumeMounts:
- name: config
mountPath: /etc/coredns
volumes:
- name: config
configMap:
name: dns-test
EOF
k -n tag08 apply -f dns-test.yaml
k -n tag08 rollout status deploy dns-test
cat > kaputt.Corefile <<'EOF'
.:53 {
errors
log
health {
lameduck 5s
ready
hosts {
10.20.30.40 test.lab.local
}
forward . /etc/resolv.conf
cache 30
reload
}
EOF
k -n tag08 create configmap dns-test --from-file=Corefile=kaputt.Corefile $do \
| k replace -f -Wait about two minutes, then run
k -n tag08 rollout restart deployment dns-test.
Hint: First read the old Pod’s logs, then run
coredns -conf against the file – and afterwards test both
names with dig @<Pod-IP>.
Solution
k -n tag08 logs deploy/dns-test | grep -E 'Reload|\[ERROR\]'
k -n tag08 get pods -l app=dns-test
coredns -conf kaputt.Corefile 2>&1 | tail -1Expected (abridged):
[INFO] Reloading
[ERROR] Restart failed: plugin/health: /etc/coredns/Corefile:6 - Error during parsing:
unknown property 'ready'
[ERROR] plugin/reload: Corefile changed but reload failed: ...
NAME READY STATUS RESTARTS AGE
dns-test-57c59bdf4b-w9hsg 0/1 CrashLoopBackOff 2 (19s ago) 40s
dns-test-7b85dbcd85-z47s8 1/1 Running 0 3m14s
plugin/health: kaputt.Corefile:6 - Error during parsing: unknown property 'ready'
The running Pod rejected the broken Corefile and
keeps working with the old configuration – that’s why the error often
surfaces only at the next restart or node failure. The new Pod crashes,
and the rolling update keeps the old one alive. The closing brace after
lameduck 5s is missing:
sed -i 's/^ lameduck 5s$/&\n }/' kaputt.Corefile
coredns -conf kaputt.Corefile 2>&1 | tail -1
k -n tag08 create configmap dns-test --from-file=Corefile=kaputt.Corefile $do \
| k replace -f -
k -n tag08 rollout status deploy dns-test --timeout=180s
IP=$(k -n tag08 get pod -l app=dns-test -o jsonpath='{.items[0].status.podIP}')
k -n tag08 exec client -- dig +short @$IP test.lab.local
k -n tag08 exec client -- dig @$IP example.com | grep statusExpected (abridged):
Listen: listen tcp :53: bind: permission denied
configmap/dns-test replaced
deployment "dns-test" successfully rolled out
10.20.30.40
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 44441
permission denied means: the syntax is fine, only port
53 requires root. That leaves the second error: hosts
without fallthrough passes nothing on to
forward.
sed -i 's/^ 10.20.30.40 test.lab.local$/&\n fallthrough/' kaputt.Corefile
k -n tag08 create configmap dns-test --from-file=Corefile=kaputt.Corefile $do \
| k replace -f -
k -n tag08 logs deploy/dns-test -f --tail=0
k -n tag08 exec client -- dig +short @$IP example.comExpected (abridged; stop logs -f with
Ctrl+C after Reloading complete):
[INFO] Reloading
[INFO] Reloading complete
172.66.147.243
104.20.23.154
Save your progress? Sign up for free
Cleanup
Restore the original Corefile exactly (remove
resourceVersion, otherwise replace rejects the
old version) and compare:
grep -v 'resourceVersion:' ~/ckne/tag08/coredns-vorher.yaml | k replace -f -
diff <(k -n kube-system get cm coredns -o jsonpath='{.data}') \
<(k create -f ~/ckne/tag08/coredns-vorher.yaml $do -o jsonpath='{.data}') \
&& echo IDENTISCHExpected:
configmap/coredns replaced
IDENTISCH
After approx. one minute, both Pods again have a hash different from the one after Drill 3 (that of the original Corefile):
k -n kube-system logs -l k8s-app=kube-dns --since=3m --tail=-1 --prefix | grep SHA512
k -n tag08 exec client -- dig +short db.lab.intern web.tag08.svc.cluster.local \
example.comExpected (abridged; db.lab.intern no
longer returns anything):
[pod/coredns-559f6c778d-vwrtf/coredns] [INFO] plugin/reload: Running configuration
SHA512 = 1b226df79860026c...
[pod/coredns-559f6c778d-6png2/coredns] [INFO] plugin/reload: Running configuration
SHA512 = 1b226df79860026c...
10.109.153.18
172.66.147.243
104.20.23.154
Remove the DNS server on the router, delete the namespace:
ssh [email protected] "sudo systemctl stop tag08-dns"
ssh [email protected] "sudo rm -rf /etc/coredns /usr/local/bin/coredns"
dig +short +time=2 +tries=1 @172.16.0.123 db.lab.intern
k delete namespace tag08Expected: connection refused from the
router, namespace "tag08" deleted. The unit
tag08-dns disappears by itself when stopped. The
coredns binary on cp1 stays as a checking
tool; so do the distributed CoreDNS replicas from Drill 7.
Exam Tip
- In the exam, changing the Corefile usually means
k -n kube-system edit cm coredns. Then wait approx. 1 min and watch the logs forReloading complete– ork -n kube-system rollout restart deploy corednswhen time is short. - Always test from a Pod and give
digthe full name (dig doesn’t use the search list). Ask a CoreDNS Pod directly first, then 10.96.0.10. - Docs: kubernetes.io “Customizing DNS Service” (stub domains, upstream) and “DNS for Services and Pods” (names, dnsPolicy, dnsConfig); coredns.io/plugins for options.
Enterprise Note
- The Corefile belongs in Git;
kubeadm upgrademigrates the ConfigMap, while some distributions and managed offerings overwrite it – they have their own extension points (such as an additionalcoredns-customConfigMap). - Scale CoreDNS with HPA or cluster-proportional-autoscaler, with a
PDB and strict distribution across zones; lowering
ndotsper application often saves half the load. - NodeLocal DNSCache (with Cilium via Local Redirect Policy) is standard in large clusters – lower latency, no UDP conntrack bottleneck.
- Metrics on :9153 (
coredns_dns_responses_totalby rcode,coredns_reload_failed_total) belong in monitoring – a high NXDOMAIN share points to ndots search chains. - In production,
loggenerates large volumes of logs: enable it only selectively and temporarily, or filter by class (log . { class denial error }).