Skip to content
Hands-on IT IT Drills

Day 8: Customizing CoreDNS

Goal: You read and change the Corefile safely, understand Service and Pod names, ndots and dnsConfig, and find DNS errors quickly. · Duration: approx. 90 min · Prerequisite: Day 6

Briefing

  • Curriculum: “CKNE: Service Networking and DNS” (Customizing CoreDNS) and “Troubleshooting Pod Connectivity” (DNS).
  • CoreDNS is a perfectly ordinary Deployment behind the Service kube-dns (10.96.0.10); its configuration lives in the ConfigMap coredns.
  • The kubelet writes nameserver 10.96.0.10, the search list and ndots:5 into every Pod’s /etc/resolv.conf.
  • The reload plugin picks up Corefile changes without restarting Pods – a typo can still take down cluster DNS. That’s why today’s rule is: check first, then apply.
 Pod (resolv.conf: 10.96.0.10, ndots:5)
   |  Cilium Socket-LB: 10.96.0.10:53 -> CoreDNS Pod
   v
 CoreDNS (ConfigMap coredns, Corefile)
   |-- cluster.local      -> kubernetes plugin (Services, Pods)
   |-- lab.intern         -> forward 172.16.0.123 (router, Drill 6)
   '-- everything else    -> forward the node's /etc/resolv.conf

Drill 1: Read the CoreDNS setup

First you back up the original ConfigMap – at the end of the day you restore exactly this.

mkdir -p ~/ckne/tag08 && cd ~/ckne/tag08
k -n kube-system get cm coredns -o yaml > ~/ckne/tag08/coredns-vorher.yaml
k -n kube-system get deploy,svc,endpointslices -l k8s-app=kube-dns

Expected (abridged):

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   2/2     2            2           41m

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   41m

NAME             ADDRESSTYPE   PORTS        ENDPOINTS              AGE
kube-dns-jjx5f   IPv4          53,53,9153   10.0.0.219,10.0.0.71   41m

For historical reasons the Service is called kube-dns; CoreDNS runs behind it. How does a Pod know the IP? From the kubelet configuration:

sudo grep -A1 -E 'clusterDNS|clusterDomain|resolvConf' /var/lib/kubelet/config.yaml
grep -v '^#' /run/systemd/resolve/resolv.conf

Expected (abridged):

clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
resolvConf: /run/systemd/resolve/resolv.conf
nameserver 172.16.0.15
nameserver 8.8.8.8
nameserver 1.1.1.1
search localdomain

resolvConf is the file that Pods with dnsPolicy: Default get – including CoreDNS itself (k -n kube-system get deploy coredns -o jsonpath='{..dnsPolicy}' → Default). That’s where the upstream servers for external names come from. Now the Corefile:

k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}'

Expected:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30 {
       disable success cluster.local
       disable denial cluster.local
    }
    loop
    reload
    loadbalance
}
Plugin Purpose
errors, log errors or every query, respectively, to the container log
health, ready liveness (:8080) and readiness (:8181) of the Pod
kubernetes answers cluster.local and reverse zones from the API
forward everything else to upstream servers
cache, loop, reload, loadbalance cache, loop protection, reloading, rotation

.:53 is a server block for the root zone, so for all names. The order of plugins in the block doesn’t matter – CoreDNS processes them in a fixed order compiled into the binary. Finally, a test ground:

k create namespace tag08
k -n tag08 create deployment web --image=nginx:1.29 --replicas=2
k -n tag08 expose deployment web --port=80
k -n tag08 expose deployment web --name=web-hl --cluster-ip=None --port=80
k -n tag08 run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k -n tag08 wait --for=condition=Ready pod --all --timeout=120s
k -n tag08 exec client -- cat /etc/resolv.conf

Expected:

search tag08.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5

Save your progress? Sign up for free

Drill 2: Service and Pod names

k -n tag08 get pods,svc -o wide
k -n tag08 exec client -- dig +short web.tag08.svc.cluster.local
k -n tag08 exec client -- dig +short web-hl.tag08.svc.cluster.local
k -n tag08 exec client -- dig +short web
k -n tag08 exec client -- dig +short +search web

Expected (abridged):

pod/client                 1/1   Running   0   48s   10.0.2.84    worker2
pod/web-689b9bdbdd-c2t4c   1/1   Running   0   49s   10.0.1.121   worker1
pod/web-689b9bdbdd-ch5rc   1/1   Running   0   49s   10.0.2.231   worker2
service/web      ClusterIP   10.109.153.18   <none>   80/TCP   49s   app=web
service/web-hl   ClusterIP   None            <none>   80/TCP   0s    app=web
10.109.153.18
10.0.2.231
10.0.1.121
10.109.153.18

Normal Service → one A answer with the ClusterIP; headless → the Pod IPs. dig web stays empty: dig ignores the search list; only with +search (or in nslookup, curl, applications) does web become web.tag08.svc.cluster.local.

SRV records exist only for named ports. expose doesn’t assign a name, so add one:

k -n tag08 patch svc web --type=json \
  -p '[{"op":"add","path":"/spec/ports/0/name","value":"http"}]'
k -n tag08 exec client -- dig +short SRV _http._tcp.web.tag08.svc.cluster.local
k -n tag08 exec client -- dig +short 10-0-1-121.tag08.pod.cluster.local
k -n tag08 exec client -- dig +short -x 10.109.153.18
k -n tag08 exec client -- dig +short -x 10.0.1.121

Expected:

0 100 80 web.tag08.svc.cluster.local.
10.0.1.121
web.tag08.svc.cluster.local.
10-0-1-121.web.tag08.svc.cluster.local.
10-0-1-121.web-hl.tag08.svc.cluster.local.

Because of pods insecure, CoreDNS returns Pod A records (<ip-with-dashes>.<ns>.pod.cluster.local) for any IP, without checking whether the Pod exists. You get stable Pod names via hostname and subdomain plus a headless Service:

k -n tag08 apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: db
  labels:
    app: web
spec:
  hostname: db
  subdomain: web-hl
  containers:
  - name: nginx
    image: nginx:1.29
EOF
k -n tag08 wait --for=condition=Ready pod/db --timeout=60s
k -n tag08 exec client -- dig +short db.web-hl.tag08.svc.cluster.local
k -n tag08 exec db -- hostname -f

Expected:

10.0.1.198
db.web-hl.tag08.svc.cluster.local

This is also how StatefulSet Pods get their names (web-0.<service>…): subdomain must match the name of a headless Service that selects the Pod.

Save your progress? Sign up for free

Drill 3: Watch the log plugin and reload

Before you change the Corefile, you fetch CoreDNS as a binary onto cp1 – in the same version as the image. That lets you check the syntax before the cluster sees it. First read the version from the Deployment:

k -n kube-system get deploy coredns -o jsonpath='{..image}'; echo

Expected: registry.k8s.io/coredns/coredns:v1.14.6 – hence V=1.14.6:

cd /tmp
V=1.14.6
ARCH=$(dpkg --print-architecture)
U=https://github.com/coredns/coredns/releases/download/v$V
curl -sSLO $U/coredns_${V}_linux_${ARCH}.tgz
curl -sSLO $U/coredns_${V}_linux_${ARCH}.tgz.sha256
sha256sum -c coredns_${V}_linux_${ARCH}.tgz.sha256
sudo tar xzf coredns_${V}_linux_${ARCH}.tgz --no-same-owner -C /usr/local/bin
coredns -version
cd ~/ckne/tag08

Expected:

coredns_1.14.6_linux_amd64.tgz: OK
CoreDNS-1.14.6
linux/amd64, go1.26.5, 424d125

Fetch the Corefile, insert log after errors, check it:

k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
sed -i 's/^    errors$/    errors\n    log/' Corefile
head -3 Corefile
coredns -conf Corefile

Expected:

.:53 {
    errors
    log
maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
plugin/kubernetes: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and
KUBERNETES_SERVICE_PORT must be defined

Here this message is the good result: the Corefile is syntactically fine, only the kubernetes plugin finds no API outside the cluster. A typo would look like this: Corefile:13 - Error during parsing: Unknown directive 'forwrd'. Limitation: errors in plugins that CoreDNS sets up only after kubernetes (such as arguments of forward) don’t show up here.

Apply it and watch the reload:

k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -
date +%T
k -n kube-system logs -l k8s-app=kube-dns -f --tail=0 --prefix

Expected (abridged, after approx. 1 min, stop with Ctrl+C):

configmap/coredns replaced
23:19:45
[pod/coredns-559f6c778d-nz7t6/coredns] [INFO] Reloading
[pod/coredns-559f6c778d-nz7t6/coredns] [INFO] plugin/reload: Running configuration
SHA512 = 2dd49c56b94cd567...
[pod/coredns-559f6c778d-nz7t6/coredns] [INFO] Reloading complete
[pod/coredns-559f6c778d-nz7t6/coredns] [INFO] 127.0.0.1:43758 - 562 "HINFO IN
4160184936247868763.2761932811795487505. udp 57 false 512" NXDOMAIN ...

Why does this take up to a minute? The kubelet updates mounted ConfigMaps with a delay, and by default reload checks the file every 30 s (with some jitter). Each Pod reloads on its own – for a short while the two replicas answer differently. The HINFO query comes from the loop plugin, which tests for loops after every load. The kubectl edit cm coredns variant works the same way, just without a syntax check first.

Save your progress? Sign up for free

Drill 4: ndots, dnsPolicy and dnsConfig

With log you can now see what a harmless curl http://example.com costs:

CLIENT_IP=$(k -n tag08 get pod client -o jsonpath='{.status.podIP}')
k -n tag08 exec client -- curl -s -o /dev/null -w '%{http_code}\n' http://example.com
k -n kube-system logs -l k8s-app=kube-dns --since=20s --tail=-1 \
  | grep "$CLIENT_IP" | grep '"A IN'

Expected (abridged):

200
[INFO] 10.0.2.84:53716 - 59923 "A IN example.com.tag08.svc.cluster.local. ..." NXDOMAIN
[INFO] 10.0.2.84:53716 - 33823 "A IN example.com.svc.cluster.local. ..." NXDOMAIN
[INFO] 10.0.2.84:53716 - 64596 "A IN example.com.cluster.local. ..." NXDOMAIN
[INFO] 10.0.2.84:53716 - 4669 "A IN example.com.localdomain. ..." NXDOMAIN
[INFO] 10.0.2.84:53716 - 2726 "A IN example.com. udp 40 false 1232" NOERROR ...

example.com has fewer than 5 dots (ndots:5), so the resolver first tries all four search domains – with AAAA that’s 10 queries instead of 2. --tail=-1 is needed because otherwise logs -l shows only 10 lines per Pod. The per-Pod remedy: dnsConfig.

cat > client-ndots.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: client-ndots
spec:
  dnsConfig:
    options:
    - name: ndots
      value: "1"
  containers:
  - name: netshoot
    image: nicolaka/netshoot:v0.16
    command: ["sleep", "infinity"]
EOF
cat > client-none.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: client-none
spec:
  dnsPolicy: None
  dnsConfig:
    nameservers: ["1.1.1.1"]
    searches: ["example.org"]
  containers:
  - name: netshoot
    image: nicolaka/netshoot:v0.16
    command: ["sleep", "infinity"]
EOF
k -n tag08 apply -f client-ndots.yaml -f client-none.yaml
k -n tag08 wait --for=condition=Ready pod/client-ndots pod/client-none
k -n tag08 exec client-ndots -- cat /etc/resolv.conf
k -n tag08 exec client-none -- cat /etc/resolv.conf

Expected:

search tag08.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:1
search example.org
nameserver 1.1.1.1

dnsConfig is merged with the dnsPolicy; only with None does dnsConfig alone count. The effects in the log:

NIP=$(k -n tag08 get pod client-ndots -o jsonpath='{.status.podIP}')
k -n tag08 exec client-ndots -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://example.com
k -n tag08 exec client-ndots -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://web.tag08
k -n kube-system logs -l k8s-app=kube-dns --since=20s --tail=-1 \
  | grep "$NIP" | grep '"A IN'
k -n tag08 exec client-none -- curl -s -m 5 http://web.tag08.svc.cluster.local
echo $?

Expected (abridged):

200
200
[INFO] 10.0.1.178:58941 - 17739 "A IN example.com. ..." NOERROR qr,rd,ra 83 0.0104s
[INFO] 10.0.1.178:52099 - 48594 "A IN web.tag08. ..." NXDOMAIN qr,rd,ra 102 0.4078s
[INFO] 10.0.1.178:52099 - 7446 "A IN web.tag08.tag08.svc.cluster.local. ..." NXDOMAIN
[INFO] 10.0.1.178:52099 - 51482 "A IN web.tag08.svc.cluster.local. ..." NOERROR
command terminated with exit code 6
6

With ndots:1, example.com goes out immediately. The price: web.tag08 now also has a dot, so it is first queried on the internet as an absolute name (0.4 s until NXDOMAIN) and only then found via the search list. client-none doesn’t know cluster DNS at all (curl exit 6 = name could not be resolved). Without changing the Pod, a trailing dot helps: curl http://example.com. produces exactly one A query.

Save your progress? Sign up for free

Drill 5: hosts and rewrite

Two typical exam requests: a fixed name for an IP outside the cluster and an alias for a Service. Open the Corefile (vim Corefile) and insert below ready:

    hosts {
       172.16.0.123 router.lab.local
       fallthrough
    }
    rewrite name exact notes.lab.local web.tag08.svc.cluster.local
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | diff - Corefile
coredns -conf Corefile 2>&1 | tail -1
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -

Expected (abridged):

7a8,12
>     hosts {
>        172.16.0.123 router.lab.local
>        fallthrough
>     }
>     rewrite name exact notes.lab.local web.tag08.svc.cluster.local
plugin/kubernetes: unable to load in-cluster configuration, ...
configmap/coredns replaced

After about a minute (log: Reloading complete for both Pods):

k -n tag08 exec client -- dig +short router.lab.local
k -n tag08 exec client -- dig +short -x 172.16.0.123
k -n tag08 exec client -- dig notes.lab.local | grep -A1 'ANSWER SECTION'
k -n tag08 exec client -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://notes.lab.local
k -n tag08 exec client -- dig +short example.com

Expected:

172.16.0.123
router.lab.local.
;; ANSWER SECTION:
notes.lab.local.    30  IN  A   10.109.153.18
200
172.66.147.243
104.20.23.154

hosts also creates the PTR record. rewrite rewrites the question before the kubernetes plugin sees it, and rewrites the answer back – the client keeps seeing notes.lab.local. Without fallthrough, hosts answers every other name with SERVFAIL (you test that in the challenge, not on the cluster DNS).

Save your progress? Sign up for free

Drill 6: Stub domain to the router

Corporate networks have their own zones that only an internal DNS server knows. On the router you play that server – with the same CoreDNS binary:

ssh [email protected]
cd /tmp
V=1.14.6
ARCH=$(dpkg --print-architecture)
U=https://github.com/coredns/coredns/releases/download/v$V
curl -sSLO $U/coredns_${V}_linux_${ARCH}.tgz
curl -sSLO $U/coredns_${V}_linux_${ARCH}.tgz.sha256
sha256sum -c coredns_${V}_linux_${ARCH}.tgz.sha256
sudo tar xzf coredns_${V}_linux_${ARCH}.tgz --no-same-owner -C /usr/local/bin
sudo mkdir -p /etc/coredns
cat <<'EOF' | sudo tee /etc/coredns/Corefile
lab.intern:53 {
    bind 172.16.0.123
    hosts {
        172.16.0.123 router.lab.intern
        10.20.30.40  db.lab.intern
    }
    log
    errors
}
EOF
sudo systemd-run --unit=tag08-dns /usr/local/bin/coredns -conf /etc/coredns/Corefile
sudo ss -lunp | grep coredns
exit

Expected (abridged):

coredns_1.14.6_linux_amd64.tgz: OK
Running as unit: tag08-dns.service; invocation ID: 4ea3e335ed444ec5a3759c0995aea187
UNCONN 0      0           172.16.0.123:53        0.0.0.0:*    users:(("coredns",...))

bind 172.16.0.123 is needed because systemd-resolved already listens on 127.0.0.53:53. systemd-run starts a transient unit that you can stop later with one command. Back on cp1: before and after.

dig +short @172.16.0.123 db.lab.intern
k -n tag08 exec client -- dig db.lab.intern | grep status
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
cat >> Corefile <<'EOF'
lab.intern:53 {
    errors
    cache 30
    forward . 172.16.0.123
}
EOF
coredns -conf Corefile 2>&1 | tail -1
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -

Expected (abridged):

10.20.30.40
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36252
plugin/kubernetes: unable to load in-cluster configuration, ...
configmap/coredns replaced

After the reload:

k -n tag08 exec client -- dig db.lab.intern | grep -A1 -E 'status|ANSWER SEC'
k -n tag08 exec client -- dig +short router.lab.intern
ssh [email protected] "sudo journalctl -u tag08-dns --no-pager | tail -2"

Expected (abridged):

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6996
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; ANSWER SECTION:
db.lab.intern.      30  IN  A   10.20.30.40
172.16.0.123
... coredns[3576]: [INFO] 172.16.0.126:50139 - 35812 "A IN db.lab.intern. ..." NOERROR
... coredns[3576]: [INFO] 172.16.0.126:50139 - 9092 "A IN router.lab.intern. ..." ...

The longest matching server block wins: lab.intern goes to the router, everything else continues through .:53. The router sees the node IP (here cp1) as the source, because Cilium masquerades Pod traffic leaving the cluster. This is the Corefile form of the “stub domain”; the same line forward . <IP> in the .:53 block would redirect all external names (“upstream nameserver”).

Save your progress? Sign up for free

Drill 7: Scaling CoreDNS, anti-affinity

k -n kube-system get pods -l k8s-app=kube-dns -o wide
k -n kube-system get deploy coredns -o jsonpath='{.spec.template.spec.affinity}'; echo

Expected (abridged):

NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE
coredns-559f6c778d-nz7t6   1/1     Running   0          48m   10.0.0.71    cp1
coredns-559f6c778d-vwrtf   1/1     Running   0          48m   10.0.0.219   cp1
{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTer
m":{"labelSelector":...,"topologyKey":"kubernetes.io/hostname"},"weight":100}]}}

Both replicas on cp1: they were created before the workers joined, and the anti-affinity is only preferred. If cp1 fails, DNS is gone. Scaling redistributes:

k -n kube-system scale deployment coredns --replicas=3
k -n kube-system rollout status deployment coredns
k -n kube-system get pods -l k8s-app=kube-dns -o wide
k -n kube-system scale deployment coredns --replicas=2
k -n kube-system get pods -l k8s-app=kube-dns -o wide

Expected (abridged):

coredns-559f6c778d-6png2   1/1     Running   0          6s    10.0.1.49    worker1
coredns-559f6c778d-nz7t6   1/1     Running   0          48m   10.0.0.71    cp1
coredns-559f6c778d-vwrtf   1/1     Running   0          48m   10.0.0.219   cp1
deployment.apps/coredns scaled
coredns-559f6c778d-6png2   1/1     Running   0          15s   10.0.1.49    worker1
coredns-559f6c778d-vwrtf   1/1     Running   0          48m   10.0.0.219   cp1

When scaling down, the ReplicaSet controller prefers deleting Pods on nodes with several replicas – now they are spread out. The new Pod reads the current Corefile; lameduck 5s lets the deleted Pod still answer in-flight queries. For strict distribution: requiredDuringScheduling… or topologySpreadConstraints, plus a PodDisruptionBudget.

Save your progress? Sign up for free

Drill 8: NodeLocal DNSCache and Cilium

NodeLocal DNSCache (kubernetes.io: “Using NodeLocal DNSCache in Kubernetes Clusters”) is a DaemonSet with a CoreDNS cache on every node. Classically it listens on a link-local IP (169.254.20.10) and uses iptables to also intercept packets to the kube-dns IP. Advantages: lower latency, no conntrack problems with UDP, less load on CoreDNS.

With Cilium KPR this trick doesn’t work: the Socket-LB translates 10.96.0.10 into a CoreDNS Pod IP already at connect() in the Pod, so the iptables rules never see the Service IP. That’s why docs.cilium.io describes its own approach under “Local Redirect Policy” → “Node-local DNS cache”: node-local-dns as a normal Pod (hostNetwork: false, -setupiptables=false) plus a CiliumLocalRedirectPolicy that redirects the Service kube-dns to the cache Pod on the same node. It requires the Helm value localRedirectPolicies.enabled=true – disabled in this lab:

cilium config view | grep -ci redirect
k get crd | grep -c ciliumlocalredirectpolicies

Expected:

0
0

The policy from the Cilium example (examples/kubernetes-local-redirect/) looks like this – read only, don’t apply:

apiVersion: cilium.io/v2
kind: CiliumLocalRedirectPolicy
metadata:
  name: nodelocaldns
  namespace: kube-system
spec:
  redirectFrontend:
    serviceMatcher:
      serviceName: kube-dns
      namespace: kube-system
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        k8s-app: node-local-dns
    toPorts:
    - port: "53"
      name: dns
      protocol: UDP
    - port: "53"
      name: dns-tcp
      protocol: TCP

Afterwards, cilium-dbg service list shows the type LocalRedirect for 10.96.0.10 with only local backends. If the cache Pod is missing, Cilium falls back to the normal CoreDNS backends.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the cluster DNS IP and cluster domain that the kubelet on cp1 gives to Pods. Target time: 20 s
  2. In tag08, start the Pod fqdn (netshoot) with ndots:2. Target time: 2 min
  3. Resolve the Pod DNS name of a web Pod. Target time: 1 min
  4. Show which upstream servers CoreDNS forwards external names to. Target time: 1 min
  5. Check from Cilium’s point of view whether kube-dns has backends. Target time: 30 s

Solution

sudo grep -A1 -E 'clusterDNS|clusterDomain' /var/lib/kubelet/config.yaml
k -n tag08 run fqdn --image=nicolaka/netshoot:v0.16 $do -- sleep infinity > fqdn.yaml
vim fqdn.yaml

Add below dnsPolicy: ClusterFirst:

  dnsConfig:
    options:
    - name: ndots
      value: "2"
k apply -f fqdn.yaml
k -n tag08 exec fqdn -- grep options /etc/resolv.conf
P=$(k -n tag08 get pod -l app=web,pod-template-hash \
  -o jsonpath='{.items[0].status.podIP}')
k -n tag08 exec client -- dig +short ${P//./-}.tag08.pod.cluster.local
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep forward
sudo grep resolvConf /var/lib/kubelet/config.yaml
grep ^nameserver /run/systemd/resolve/resolv.conf
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A2 '10.96.0.10:53/UDP'

Expected (abridged):

clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
options ndots:2
10.0.1.121
    forward . /etc/resolv.conf {
resolvConf: /run/systemd/resolve/resolv.conf
nameserver 172.16.0.15
nameserver 8.8.8.8
nameserver 1.1.1.1
5    10.96.0.10:53/UDP         ClusterIP      1 => 10.0.0.219:53/UDP (active)
                                              2 => 10.0.1.49:53/UDP (active)

/etc/resolv.conf in the forward entry is the file inside the CoreDNS Pod – because of dnsPolicy: Default, the node’s resolvConf file. For network problems, ask a CoreDNS Pod directly (dig @10.0.0.219 …) to separate the Service path from CoreDNS.

Save your progress? Sign up for free

Checkpoint

k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' \
  | grep -E 'log|hosts|rewrite|lab.intern'

Expected: the lines log, hosts {, rewrite name exact … and lab.intern:53 {.

k -n tag08 exec client -- dig +short router.lab.local

Expected: 172.16.0.123

k -n tag08 exec client -- dig +short notes.lab.local

Expected: the ClusterIP of web (here 10.109.153.18).

k -n tag08 exec client -- dig +short db.lab.intern

Expected: 10.20.30.40

k -n tag08 exec client-ndots -- grep options /etc/resolv.conf

Expected: options ndots:1

k -n kube-system get pods -l k8s-app=kube-dns -o wide

Expected: two CoreDNS Pods Running on different nodes.

Challenge

A copy of CoreDNS (dns-test in tag08) no longer starts after a restart, and even once repaired it doesn’t answer external names. Get it running without touching the cluster DNS: test.lab.local → 10.20.30.40, example.com resolvable. Preparation (first the working copy, then the broken configuration):

cat > dns-test.Corefile <<'EOF'
.:53 {
    errors
    log
    health {
       lameduck 5s
    }
    ready
    hosts {
       10.20.30.40 test.lab.local
       fallthrough
    }
    forward . /etc/resolv.conf
    cache 30
    reload
}
EOF
k -n tag08 create configmap dns-test --from-file=Corefile=dns-test.Corefile
cat > dns-test.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dns-test
  template:
    metadata:
      labels:
        app: dns-test
    spec:
      containers:
      - name: coredns
        image: registry.k8s.io/coredns/coredns:v1.14.6
        args: ["-conf", "/etc/coredns/Corefile"]
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
        volumeMounts:
        - name: config
          mountPath: /etc/coredns
      volumes:
      - name: config
        configMap:
          name: dns-test
EOF
k -n tag08 apply -f dns-test.yaml
k -n tag08 rollout status deploy dns-test
cat > kaputt.Corefile <<'EOF'
.:53 {
    errors
    log
    health {
       lameduck 5s
    ready
    hosts {
       10.20.30.40 test.lab.local
    }
    forward . /etc/resolv.conf
    cache 30
    reload
}
EOF
k -n tag08 create configmap dns-test --from-file=Corefile=kaputt.Corefile $do \
  | k replace -f -

Wait about two minutes, then run k -n tag08 rollout restart deployment dns-test.

Hint: First read the old Pod’s logs, then run coredns -conf against the file – and afterwards test both names with dig @<Pod-IP>.

Solution

k -n tag08 logs deploy/dns-test | grep -E 'Reload|\[ERROR\]'
k -n tag08 get pods -l app=dns-test
coredns -conf kaputt.Corefile 2>&1 | tail -1

Expected (abridged):

[INFO] Reloading
[ERROR] Restart failed: plugin/health: /etc/coredns/Corefile:6 - Error during parsing:
unknown property 'ready'
[ERROR] plugin/reload: Corefile changed but reload failed: ...
NAME                        READY   STATUS             RESTARTS      AGE
dns-test-57c59bdf4b-w9hsg   0/1     CrashLoopBackOff   2 (19s ago)   40s
dns-test-7b85dbcd85-z47s8   1/1     Running            0             3m14s
plugin/health: kaputt.Corefile:6 - Error during parsing: unknown property 'ready'

The running Pod rejected the broken Corefile and keeps working with the old configuration – that’s why the error often surfaces only at the next restart or node failure. The new Pod crashes, and the rolling update keeps the old one alive. The closing brace after lameduck 5s is missing:

sed -i 's/^       lameduck 5s$/&\n    }/' kaputt.Corefile
coredns -conf kaputt.Corefile 2>&1 | tail -1
k -n tag08 create configmap dns-test --from-file=Corefile=kaputt.Corefile $do \
  | k replace -f -
k -n tag08 rollout status deploy dns-test --timeout=180s
IP=$(k -n tag08 get pod -l app=dns-test -o jsonpath='{.items[0].status.podIP}')
k -n tag08 exec client -- dig +short @$IP test.lab.local
k -n tag08 exec client -- dig @$IP example.com | grep status

Expected (abridged):

Listen: listen tcp :53: bind: permission denied
configmap/dns-test replaced
deployment "dns-test" successfully rolled out
10.20.30.40
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 44441

permission denied means: the syntax is fine, only port 53 requires root. That leaves the second error: hosts without fallthrough passes nothing on to forward.

sed -i 's/^       10.20.30.40 test.lab.local$/&\n       fallthrough/' kaputt.Corefile
k -n tag08 create configmap dns-test --from-file=Corefile=kaputt.Corefile $do \
  | k replace -f -
k -n tag08 logs deploy/dns-test -f --tail=0
k -n tag08 exec client -- dig +short @$IP example.com

Expected (abridged; stop logs -f with Ctrl+C after Reloading complete):

[INFO] Reloading
[INFO] Reloading complete
172.66.147.243
104.20.23.154

Save your progress? Sign up for free

Cleanup

Restore the original Corefile exactly (remove resourceVersion, otherwise replace rejects the old version) and compare:

grep -v 'resourceVersion:' ~/ckne/tag08/coredns-vorher.yaml | k replace -f -
diff <(k -n kube-system get cm coredns -o jsonpath='{.data}') \
  <(k create -f ~/ckne/tag08/coredns-vorher.yaml $do -o jsonpath='{.data}') \
  && echo IDENTISCH

Expected:

configmap/coredns replaced
IDENTISCH

After approx. one minute, both Pods again have a hash different from the one after Drill 3 (that of the original Corefile):

k -n kube-system logs -l k8s-app=kube-dns --since=3m --tail=-1 --prefix | grep SHA512
k -n tag08 exec client -- dig +short db.lab.intern web.tag08.svc.cluster.local \
  example.com

Expected (abridged; db.lab.intern no longer returns anything):

[pod/coredns-559f6c778d-vwrtf/coredns] [INFO] plugin/reload: Running configuration
SHA512 = 1b226df79860026c...
[pod/coredns-559f6c778d-6png2/coredns] [INFO] plugin/reload: Running configuration
SHA512 = 1b226df79860026c...
10.109.153.18
172.66.147.243
104.20.23.154

Remove the DNS server on the router, delete the namespace:

ssh [email protected] "sudo systemctl stop tag08-dns"
ssh [email protected] "sudo rm -rf /etc/coredns /usr/local/bin/coredns"
dig +short +time=2 +tries=1 @172.16.0.123 db.lab.intern
k delete namespace tag08

Expected: connection refused from the router, namespace "tag08" deleted. The unit tag08-dns disappears by itself when stopped. The coredns binary on cp1 stays as a checking tool; so do the distributed CoreDNS replicas from Drill 7.

Exam Tip

  • In the exam, changing the Corefile usually means k -n kube-system edit cm coredns. Then wait approx. 1 min and watch the logs for Reloading complete – or k -n kube-system rollout restart deploy coredns when time is short.
  • Always test from a Pod and give dig the full name (dig doesn’t use the search list). Ask a CoreDNS Pod directly first, then 10.96.0.10.
  • Docs: kubernetes.io “Customizing DNS Service” (stub domains, upstream) and “DNS for Services and Pods” (names, dnsPolicy, dnsConfig); coredns.io/plugins for options.

Enterprise Note

  • The Corefile belongs in Git; kubeadm upgrade migrates the ConfigMap, while some distributions and managed offerings overwrite it – they have their own extension points (such as an additional coredns-custom ConfigMap).
  • Scale CoreDNS with HPA or cluster-proportional-autoscaler, with a PDB and strict distribution across zones; lowering ndots per application often saves half the load.
  • NodeLocal DNSCache (with Cilium via Local Redirect Policy) is standard in large clusters – lower latency, no UDP conntrack bottleneck.
  • Metrics on :9153 (coredns_dns_responses_total by rcode, coredns_reload_failed_total) belong in monitoring – a high NXDOMAIN share points to ndots search chains.
  • In production, log generates large volumes of logs: enable it only selectively and temporarily, or filter by class (log . { class denial error }).