Skip to content
Hands-on IT IT Drills

Day 3: Services and DNS

Goal: Put the api behind a Service with a stable name, understand DNS in the cluster, bring web to the Mac via NodePort and add the worker. · Duration: approx. 75 min · Prerequisite: Day 2

Briefing

  • Pod IPs change with every new Pod. A Service gives all Pods with a matching label a stable virtual IP (ClusterIP) and a DNS name.
  • Which Pods are currently behind it is stored in EndpointSlices. kube-proxy on every node distributes new connections across these Pods.
  • CoreDNS resolves api in your own namespace; the full name of the Service is api.notes.svc.cluster.local.
  • A NodePort Service opens a port (30000–32767) on every node. kind forwards 30080 to localhost:30080 on the Mac (Day 0).
 Mac: curl localhost:30080
        │  extraPortMappings (Day 0)
        ▼
 Node :30080 ──► Service web (NodePort) ──► Pods web ×2
                                              │ http://api:8080
                                              ▼
                 Service api (ClusterIP) ──► Pods api ×3
                                              ▲
                                  Pod worker ─┘

Drill 1: Start of the Day, Wandering IPs

source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag03 && cd ~/k8s-lab/tag03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag02/notes/." .
kubectl get pods -o custom-columns='NAME:.metadata.name,IP:.status.podIP'
kubectl delete "$(kubectl get pods -o name | head -1)"
sleep 3
kubectl get pods -o custom-columns='NAME:.metadata.name,IP:.status.podIP'

Expected (abridged, names, IPs and times will differ on your machine):

NAME                  IP
api-9cb44cd5b-8wq98   10.244.2.12
api-9cb44cd5b-g9qfn   10.244.2.13
api-9cb44cd5b-twrzx   10.244.1.8
pod "api-9cb44cd5b-8wq98" deleted from notes namespace
NAME                  IP
api-9cb44cd5b-6sf6d   10.244.1.10
api-9cb44cd5b-g9qfn   10.244.2.13
api-9cb44cd5b-twrzx   10.244.1.8

New Pod, new IP. Anyone who wants to reach the api needs something stable in front of it.

Save your progress? Sign up for free

Drill 2: A Service for the api

The Service goes into the same file as the Deployment, separated by ---:

cat > k8s/api.yaml <<'EOF'
# k8s/api.yaml – state at end of Day 3
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    app.kubernetes.io/part-of: notes
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: api
        app.kubernetes.io/part-of: notes
    spec:
      containers:
        - name: api
          image: notes-api:0.2.0
          env:
            - name: NOTES_API_KEY
              value: lab-api-key-123
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    app.kubernetes.io/part-of: notes
spec:
  selector:
    app.kubernetes.io/name: api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
EOF
kubectl apply --dry-run=server -f k8s/api.yaml
kubectl apply -f k8s/api.yaml
kubectl get svc api
kubectl get endpointslices -l kubernetes.io/service-name=api

Expected (abridged):

deployment.apps/api unchanged (server dry run)
service/api created (server dry run)
deployment.apps/api unchanged
service/api created
NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
api    ClusterIP   10.96.247.150   <none>        8080/TCP   0s
NAME        ADDRESSTYPE   PORTS   ENDPOINTS                            AGE
api-dtnfk   IPv4          8080    10.244.1.10,10.244.1.8,10.244.2.13   1s
  • selector picks the Pods – the same labels as in the Deployment, but the Service doesn’t know the Deployment at all.
  • port is the port of the Service, targetPort the one in the container.
  • The EndpointSlice was created by the control plane controller: exactly the three Pod IPs from Drill 1. Without a type, a Service is ClusterIP.
kubectl get endpointslice -l kubernetes.io/service-name=api -o yaml \
  | grep -E 'addresses:|- 10\.|ready:|name: api-'

Expected (abridged):

  - addresses:
    - 10.244.1.10
      ready: true
      name: api-9cb44cd5b-6sf6d
  - addresses:
    - 10.244.1.8
      ready: true
      name: api-9cb44cd5b-twrzx
…

Only Pods with ready: true get traffic. When a Pod is ready is something you define on Day 5 with a readiness probe.

Save your progress? Sign up for free

Drill 3: web and worker

Both talk to the api via its Service name. web gets a NodePort Service:

cat > k8s/web.yaml <<'EOF'
# k8s/web.yaml – state at end of Day 3
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
        app.kubernetes.io/part-of: notes
    spec:
      containers:
        - name: web
          image: notes-web:0.1.0
          env:
            - name: NOTES_API_URL
              value: http://api:8080
            - name: NOTES_API_KEY
              value: lab-api-key-123
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: notes
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: web
  ports:
    - name: http
      port: 3000
      targetPort: 3000
      nodePort: 30080
EOF
cat > k8s/worker.yaml <<'EOF'
# k8s/worker.yaml – state at end of Day 3
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  labels:
    app.kubernetes.io/name: worker
    app.kubernetes.io/part-of: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: worker
  template:
    metadata:
      labels:
        app.kubernetes.io/name: worker
        app.kubernetes.io/part-of: notes
    spec:
      containers:
        - name: worker
          image: notes-worker:0.1.0
          env:
            - name: NOTES_API_URL
              value: http://api:8080
            - name: NOTES_API_KEY
              value: lab-api-key-123
EOF
kubectl apply --dry-run=server -f k8s/web.yaml -f k8s/worker.yaml
kubectl apply -f k8s/web.yaml -f k8s/worker.yaml
kubectl rollout status deploy/web && kubectl rollout status deploy/worker
kubectl get deploy,svc

Expected (abridged):

deployment.apps/web created (server dry run)
service/web created (server dry run)
deployment.apps/worker created (server dry run)
deployment.apps/web created
service/web created
deployment.apps/worker created
…
deployment "web" successfully rolled out
deployment "worker" successfully rolled out
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/api      3/3     3            3           3m58s
deployment.apps/web      2/2     2            2           3s
deployment.apps/worker   1/1     1            1           3s

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/api   ClusterIP   10.96.247.150   <none>        8080/TCP         13s
service/web   NodePort    10.96.84.172    <none>        3000:30080/TCP   3s

worker has no Service: nobody calls it, it only calls the api. The images of web and worker are still 0.1.0 – the app version only matters for the api here.

Save your progress? Sign up for free

Drill 4: DNS in the Cluster

notes-web is based on Alpine and has a shell with wget and nslookup. That makes deploy/web your tool for requests inside the cluster:

kubectl exec deploy/web -- cat /etc/resolv.conf
kubectl exec deploy/web -- nslookup api.notes.svc.cluster.local
kubectl exec deploy/web -- wget -qO- http://api:8080/version
kubectl exec deploy/web -- wget -qO- http://api.notes.svc.cluster.local:8080/version
kubectl exec deploy/web -- wget -qO- -T 3 http://api.default:8080/version

Expected (abridged):

search notes.svc.cluster.local svc.cluster.local cluster.local …
nameserver 10.96.0.10
options ndots:5
Server:     10.96.0.10
Address:    10.96.0.10:53

Name:   api.notes.svc.cluster.local
Address: 10.96.247.150

0.2.0
0.2.0
wget: bad address 'api.default:8080'
command terminated with exit code 1
  • nameserver is the ClusterIP of CoreDNS, search appends the suffixes: api first becomes api.notes.svc.cluster.local.
  • The answer is the ClusterIP of the Service, not that of a Pod.
  • api.default searches in the namespace default – there is no api there. Across namespaces you write <service>.<namespace>.

In addition, Kubernetes sets variables in every Pod for the Services that already existed when it started:

kubectl exec deploy/web -- sh -c 'env | grep ^API_SERVICE'

Expected (order may differ):

API_SERVICE_PORT=8080
API_SERVICE_PORT_HTTP=8080
API_SERVICE_HOST=10.96.247.150

Rely on DNS, not on these variables: a Service created after the Pod is missing from them.

Warning: A throwaway Pod would be the obvious choice: kubectl run tmp --rm -i --restart=Never --image=busybox:1.37 -- wget -qO- …. With such short commands, kubectl often arrives too late and reports couldn't attach to pod/tmp, falling back to streaming logs – the output is missing or appears twice. On top of that, kubectl warns that input ends up in the container logs. For one-off requests use kubectl exec deploy/web -- wget … or curl from the Mac to the NodePort.

Save your progress? Sign up for free

Drill 5: When the Selector Doesn’t Match

The most common Service mistake: the Service exists, but no Pod matches the selector.

kubectl create service clusterip api-falsch --tcp 8080:8080
kubectl get svc api-falsch -o jsonpath='{.spec.selector}'; echo
kubectl get endpointslices -l kubernetes.io/service-name=api-falsch
kubectl exec deploy/web -- wget -qO- -T 3 http://api-falsch:8080/version
kubectl delete svc api-falsch

Expected (abridged):

service/api-falsch created
{"app":"api-falsch"}
NAME               ADDRESSTYPE   PORTS     ENDPOINTS   AGE
api-falsch-x89dd   IPv4          <unset>   <unset>     0s
wget: can't connect to remote host (10.96.56.78): Connection refused
command terminated with exit code 1
service "api-falsch" deleted from notes namespace

DNS works, the ClusterIP exists, but the EndpointSlice is empty – so the connection is refused. Remember the order when troubleshooting: Service → EndpointSlice → labels of the Pods (kubectl get pods --show-labels).

Save your progress? Sign up for free

Drill 6: From the Mac to the NodePort

curl -s localhost:30080/healthz; echo
curl -s localhost:30080/ | head -5
curl -s localhost:30080/api/notes
curl -s localhost:30080/api/notes -d '{"text":"ohne Header"}'; echo

Expected:

ok
<!doctype html>
<html lang="de">
<head><meta charset="utf-8"><title>notes</title></head>
<body>
  <h1>notes</h1>
[]
{"error":"text required"}

The path: Mac localhost:30080 → Colima → port 30080 of the container lab-control-plane → kube-proxy → one of the two web Pods → Service api → an api Pod. http://localhost:30080 also works in the browser.

Without Content-Type: application/json, curl sends form data, notes-web reads no text and the api answers text required. web adds the API key itself.

Save your progress? Sign up for free

Drill 7: Whose Notes?

Create six notes through web, then read several times:

for i in 1 2 3 4 5 6; do
  curl -s localhost:30080/api/notes -H 'Content-Type: application/json' \
    -d "{\"text\":\"Notiz $i\"}"
done
for i in 1 2 3 4 5 6; do curl -s localhost:30080/api/notes | jq -c 'map(.text)'; done

Expected (abridged, the distribution will differ on your machine):

{"id":"9c041330-…","text":"Notiz 1"}
…
{"id":"ca4bf083-…","text":"Notiz 6"}
["Notiz 2","Notiz 3","Notiz 4","Notiz 6"]
["Notiz 2","Notiz 3","Notiz 4","Notiz 6"]
["Notiz 1","Notiz 5"]
["Notiz 1","Notiz 5"]
["Notiz 1","Notiz 5"]
["Notiz 1","Notiz 5"]

Every api replica has its own store. Ask each one individually:

for ip in $(kubectl get pods -l app.kubernetes.io/name=api \
  -o jsonpath='{.items[*].status.podIP}'); do
  printf '%s ' "$ip"
  kubectl exec deploy/web -- wget -qO- "http://$ip:8080/notes" | jq -c 'map(.text)'
done
kubectl logs deploy/worker --tail 2

Expected (abridged):

10.244.1.10 []
10.244.2.13 ["Notiz 1","Notiz 5"]
10.244.1.8 ["Notiz 2","Notiz 3","Notiz 4","Notiz 6"]
summary: 0 notes, 0 words
summary: 0 notes, 0 words

The Service distributes connections, not requests. Every curl from the Mac is a new connection and lands on a random web Pod. Every web Pod keeps its connection to the api open (keep-alive) and, in the test, always talks to the same api replica; so does the worker – and its replica has no notes at all. Three replicas with in-process storage are three separate data sets. The solution is shared storage: Postgres as a StatefulSet on Day 7.

Save your progress? Sign up for free

Checkpoint

kubectl get deploy

Expected: api 3/3, web 2/2, worker 1/1.

NAME     READY   UP-TO-DATE   AVAILABLE   AGE
api      3/3     3            3           32s
web      2/2     2            2           14s
worker   1/1     1            1           14s
kubectl get endpointslices -l kubernetes.io/service-name=api

Expected: three addresses behind the Service api, one per Pod.

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                          AGE
api-rzgvm   IPv4          8080    10.244.2.6,10.244.1.5,10.244.2.7   14s
kubectl exec deploy/web -- wget -qO- http://api.notes.svc.cluster.local:8080/version

Expected: the full DNS name resolves, and the api answers with its version.

0.2.0
curl -s localhost:30080/healthz; echo

Expected: web answers via the NodePort.

ok
kubectl logs deploy/worker --tail 1

Expected: one summary: line (the numbers differ).

summary: 0 notes, 0 words

Challenge

Additionally expose the api under the name api-intern on port 80 without writing a file, and call http://api-intern/version from a web Pod. Clean up afterwards.

Hint: kubectl expose takes the selector from a Deployment; --port and --target-port may differ.

Solution

kubectl expose deploy api --name api-intern --port 80 --target-port 8080
kubectl get svc api-intern
kubectl get endpointslices -l kubernetes.io/service-name=api-intern
kubectl exec deploy/web -- wget -qO- http://api-intern/version
kubectl get svc api-intern -o jsonpath='{.spec.selector}'; echo
kubectl delete svc api-intern

Expected (abridged):

service/api-intern exposed
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
api-intern   ClusterIP   10.96.82.30   <none>        80/TCP    1s
NAME               ADDRESSTYPE   PORTS   ENDPOINTS                            AGE
api-intern-rbvmv   IPv4          8080    10.244.1.10,10.244.1.8,10.244.2.13   0s
0.2.0
{"app.kubernetes.io/name":"api"}
service "api-intern" deleted from notes namespace

The Service listens on 80, the EndpointSlice lists the container ports 8080. Two Services with the same selector are allowed; that way you can change ports or names without touching the Pods.

Save your progress? Sign up for free

Enterprise Note

  • NodePorts rarely face the outside directly. In front of them sits a load balancer (cloud or MetalLB) or a Gateway that understands HTTP (Day 11, “Gateway API hands-on”).
  • Because Services distribute connections, not requests, with keep-alive, HTTP/2 and gRPC all requests from one client land on one Pod. Remedy: an L7 proxy or a service mesh.
  • Who may reach which Service in the cluster is governed by NetworkPolicies (Day 11). Without them, every Pod talks to every other Pod.
  • ndots:5 makes short external names run through all search domains first. With many external calls, the fully qualified name with a trailing dot (api.example.com.) or a customized dnsConfig pays off.