Day 3: Services and DNS
Goal: Put the api behind a Service with a stable name, understand DNS in the cluster, bring web to the Mac via NodePort and add the worker. · Duration: approx. 75 min · Prerequisite: Day 2
Briefing
- Pod IPs change with every new Pod. A Service gives all Pods with a matching label a stable virtual IP (ClusterIP) and a DNS name.
- Which Pods are currently behind it is stored in EndpointSlices. kube-proxy on every node distributes new connections across these Pods.
- CoreDNS resolves
apiin your own namespace; the full name of the Service isapi.notes.svc.cluster.local. - A NodePort Service opens a port (30000–32767) on
every node. kind forwards 30080 to
localhost:30080on the Mac (Day 0).
Mac: curl localhost:30080
│ extraPortMappings (Day 0)
▼
Node :30080 ──► Service web (NodePort) ──► Pods web ×2
│ http://api:8080
▼
Service api (ClusterIP) ──► Pods api ×3
▲
Pod worker ─┘
Drill 1: Start of the Day, Wandering IPs
source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag03 && cd ~/k8s-lab/tag03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag02/notes/." .
kubectl get pods -o custom-columns='NAME:.metadata.name,IP:.status.podIP'
kubectl delete "$(kubectl get pods -o name | head -1)"
sleep 3
kubectl get pods -o custom-columns='NAME:.metadata.name,IP:.status.podIP'Expected (abridged, names, IPs and times will differ on your machine):
NAME IP
api-9cb44cd5b-8wq98 10.244.2.12
api-9cb44cd5b-g9qfn 10.244.2.13
api-9cb44cd5b-twrzx 10.244.1.8
pod "api-9cb44cd5b-8wq98" deleted from notes namespace
NAME IP
api-9cb44cd5b-6sf6d 10.244.1.10
api-9cb44cd5b-g9qfn 10.244.2.13
api-9cb44cd5b-twrzx 10.244.1.8
New Pod, new IP. Anyone who wants to reach the api needs something stable in front of it.
Save your progress? Sign up for free
Drill 2: A Service for the api
The Service goes into the same file as the Deployment, separated by
---:
cat > k8s/api.yaml <<'EOF'
# k8s/api.yaml – state at end of Day 3
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
labels:
app.kubernetes.io/name: api
app.kubernetes.io/part-of: notes
spec:
replicas: 3
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app.kubernetes.io/name: api
template:
metadata:
labels:
app.kubernetes.io/name: api
app.kubernetes.io/part-of: notes
spec:
containers:
- name: api
image: notes-api:0.2.0
env:
- name: NOTES_API_KEY
value: lab-api-key-123
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: api
labels:
app.kubernetes.io/name: api
app.kubernetes.io/part-of: notes
spec:
selector:
app.kubernetes.io/name: api
ports:
- name: http
port: 8080
targetPort: 8080
EOF
kubectl apply --dry-run=server -f k8s/api.yaml
kubectl apply -f k8s/api.yaml
kubectl get svc api
kubectl get endpointslices -l kubernetes.io/service-name=apiExpected (abridged):
deployment.apps/api unchanged (server dry run)
service/api created (server dry run)
deployment.apps/api unchanged
service/api created
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api ClusterIP 10.96.247.150 <none> 8080/TCP 0s
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
api-dtnfk IPv4 8080 10.244.1.10,10.244.1.8,10.244.2.13 1s
selectorpicks the Pods – the same labels as in the Deployment, but the Service doesn’t know the Deployment at all.portis the port of the Service,targetPortthe one in the container.- The EndpointSlice was created by the control plane
controller: exactly the three Pod IPs from Drill 1. Without a
type, a Service isClusterIP.
kubectl get endpointslice -l kubernetes.io/service-name=api -o yaml \
| grep -E 'addresses:|- 10\.|ready:|name: api-'Expected (abridged):
- addresses:
- 10.244.1.10
ready: true
name: api-9cb44cd5b-6sf6d
- addresses:
- 10.244.1.8
ready: true
name: api-9cb44cd5b-twrzx
…
Only Pods with ready: true get traffic. When a Pod is
ready is something you define on Day 5 with a readiness probe.
Save your progress? Sign up for free
Drill 3: web and worker
Both talk to the api via its Service name. web gets a NodePort Service:
cat > k8s/web.yaml <<'EOF'
# k8s/web.yaml – state at end of Day 3
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
labels:
app.kubernetes.io/name: web
app.kubernetes.io/part-of: notes
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: web
template:
metadata:
labels:
app.kubernetes.io/name: web
app.kubernetes.io/part-of: notes
spec:
containers:
- name: web
image: notes-web:0.1.0
env:
- name: NOTES_API_URL
value: http://api:8080
- name: NOTES_API_KEY
value: lab-api-key-123
ports:
- containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
name: web
labels:
app.kubernetes.io/name: web
app.kubernetes.io/part-of: notes
spec:
type: NodePort
selector:
app.kubernetes.io/name: web
ports:
- name: http
port: 3000
targetPort: 3000
nodePort: 30080
EOF
cat > k8s/worker.yaml <<'EOF'
# k8s/worker.yaml – state at end of Day 3
apiVersion: apps/v1
kind: Deployment
metadata:
name: worker
labels:
app.kubernetes.io/name: worker
app.kubernetes.io/part-of: notes
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: worker
template:
metadata:
labels:
app.kubernetes.io/name: worker
app.kubernetes.io/part-of: notes
spec:
containers:
- name: worker
image: notes-worker:0.1.0
env:
- name: NOTES_API_URL
value: http://api:8080
- name: NOTES_API_KEY
value: lab-api-key-123
EOF
kubectl apply --dry-run=server -f k8s/web.yaml -f k8s/worker.yaml
kubectl apply -f k8s/web.yaml -f k8s/worker.yaml
kubectl rollout status deploy/web && kubectl rollout status deploy/worker
kubectl get deploy,svcExpected (abridged):
deployment.apps/web created (server dry run)
service/web created (server dry run)
deployment.apps/worker created (server dry run)
deployment.apps/web created
service/web created
deployment.apps/worker created
…
deployment "web" successfully rolled out
deployment "worker" successfully rolled out
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/api 3/3 3 3 3m58s
deployment.apps/web 2/2 2 2 3s
deployment.apps/worker 1/1 1 1 3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/api ClusterIP 10.96.247.150 <none> 8080/TCP 13s
service/web NodePort 10.96.84.172 <none> 3000:30080/TCP 3s
worker has no Service: nobody calls it, it only calls the api. The images of web and worker are still 0.1.0 – the app version only matters for the api here.
Save your progress? Sign up for free
Drill 4: DNS in the Cluster
notes-web is based on Alpine and has a shell with wget
and nslookup. That makes deploy/web your tool
for requests inside the cluster:
kubectl exec deploy/web -- cat /etc/resolv.conf
kubectl exec deploy/web -- nslookup api.notes.svc.cluster.local
kubectl exec deploy/web -- wget -qO- http://api:8080/version
kubectl exec deploy/web -- wget -qO- http://api.notes.svc.cluster.local:8080/version
kubectl exec deploy/web -- wget -qO- -T 3 http://api.default:8080/versionExpected (abridged):
search notes.svc.cluster.local svc.cluster.local cluster.local …
nameserver 10.96.0.10
options ndots:5
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: api.notes.svc.cluster.local
Address: 10.96.247.150
0.2.0
0.2.0
wget: bad address 'api.default:8080'
command terminated with exit code 1
nameserveris the ClusterIP of CoreDNS,searchappends the suffixes:apifirst becomesapi.notes.svc.cluster.local.- The answer is the ClusterIP of the Service, not that of a Pod.
api.defaultsearches in the namespacedefault– there is no api there. Across namespaces you write<service>.<namespace>.
In addition, Kubernetes sets variables in every Pod for the Services that already existed when it started:
kubectl exec deploy/web -- sh -c 'env | grep ^API_SERVICE'Expected (order may differ):
API_SERVICE_PORT=8080
API_SERVICE_PORT_HTTP=8080
API_SERVICE_HOST=10.96.247.150
Rely on DNS, not on these variables: a Service created after the Pod is missing from them.
Warning: A throwaway Pod would be the obvious choice:
kubectl run tmp --rm -i --restart=Never --image=busybox:1.37 -- wget -qO- …. With such short commands, kubectl often arrives too late and reportscouldn't attach to pod/tmp, falling back to streaming logs– the output is missing or appears twice. On top of that, kubectl warns that input ends up in the container logs. For one-off requests usekubectl exec deploy/web -- wget …orcurlfrom the Mac to the NodePort.
Save your progress? Sign up for free
Drill 5: When the Selector Doesn’t Match
The most common Service mistake: the Service exists, but no Pod matches the selector.
kubectl create service clusterip api-falsch --tcp 8080:8080
kubectl get svc api-falsch -o jsonpath='{.spec.selector}'; echo
kubectl get endpointslices -l kubernetes.io/service-name=api-falsch
kubectl exec deploy/web -- wget -qO- -T 3 http://api-falsch:8080/version
kubectl delete svc api-falschExpected (abridged):
service/api-falsch created
{"app":"api-falsch"}
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
api-falsch-x89dd IPv4 <unset> <unset> 0s
wget: can't connect to remote host (10.96.56.78): Connection refused
command terminated with exit code 1
service "api-falsch" deleted from notes namespace
DNS works, the ClusterIP exists, but the EndpointSlice is empty – so
the connection is refused. Remember the order when troubleshooting:
Service → EndpointSlice → labels of the Pods
(kubectl get pods --show-labels).
Save your progress? Sign up for free
Drill 6: From the Mac to the NodePort
curl -s localhost:30080/healthz; echo
curl -s localhost:30080/ | head -5
curl -s localhost:30080/api/notes
curl -s localhost:30080/api/notes -d '{"text":"ohne Header"}'; echoExpected:
ok
<!doctype html>
<html lang="de">
<head><meta charset="utf-8"><title>notes</title></head>
<body>
<h1>notes</h1>
[]
{"error":"text required"}
The path: Mac localhost:30080 → Colima → port 30080 of
the container lab-control-plane → kube-proxy → one of the
two web Pods → Service api → an api Pod.
http://localhost:30080 also works in the browser.
Without Content-Type: application/json, curl sends form
data, notes-web reads no text and the api answers
text required. web adds the API key itself.
Save your progress? Sign up for free
Drill 7: Whose Notes?
Create six notes through web, then read several times:
for i in 1 2 3 4 5 6; do
curl -s localhost:30080/api/notes -H 'Content-Type: application/json' \
-d "{\"text\":\"Notiz $i\"}"
done
for i in 1 2 3 4 5 6; do curl -s localhost:30080/api/notes | jq -c 'map(.text)'; doneExpected (abridged, the distribution will differ on your machine):
{"id":"9c041330-…","text":"Notiz 1"}
…
{"id":"ca4bf083-…","text":"Notiz 6"}
["Notiz 2","Notiz 3","Notiz 4","Notiz 6"]
["Notiz 2","Notiz 3","Notiz 4","Notiz 6"]
["Notiz 1","Notiz 5"]
["Notiz 1","Notiz 5"]
["Notiz 1","Notiz 5"]
["Notiz 1","Notiz 5"]
Every api replica has its own store. Ask each one individually:
for ip in $(kubectl get pods -l app.kubernetes.io/name=api \
-o jsonpath='{.items[*].status.podIP}'); do
printf '%s ' "$ip"
kubectl exec deploy/web -- wget -qO- "http://$ip:8080/notes" | jq -c 'map(.text)'
done
kubectl logs deploy/worker --tail 2Expected (abridged):
10.244.1.10 []
10.244.2.13 ["Notiz 1","Notiz 5"]
10.244.1.8 ["Notiz 2","Notiz 3","Notiz 4","Notiz 6"]
summary: 0 notes, 0 words
summary: 0 notes, 0 words
The Service distributes connections, not requests. Every curl from the Mac is a new connection and lands on a random web Pod. Every web Pod keeps its connection to the api open (keep-alive) and, in the test, always talks to the same api replica; so does the worker – and its replica has no notes at all. Three replicas with in-process storage are three separate data sets. The solution is shared storage: Postgres as a StatefulSet on Day 7.
Save your progress? Sign up for free
Checkpoint
kubectl get deployExpected: api 3/3, web 2/2, worker
1/1.
NAME READY UP-TO-DATE AVAILABLE AGE
api 3/3 3 3 32s
web 2/2 2 2 14s
worker 1/1 1 1 14s
kubectl get endpointslices -l kubernetes.io/service-name=apiExpected: three addresses behind the Service api, one
per Pod.
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
api-rzgvm IPv4 8080 10.244.2.6,10.244.1.5,10.244.2.7 14s
kubectl exec deploy/web -- wget -qO- http://api.notes.svc.cluster.local:8080/versionExpected: the full DNS name resolves, and the api answers with its version.
0.2.0
curl -s localhost:30080/healthz; echoExpected: web answers via the NodePort.
ok
kubectl logs deploy/worker --tail 1Expected: one summary: line (the numbers differ).
summary: 0 notes, 0 words
Challenge
Additionally expose the api under the name api-intern on
port 80 without writing a file, and call
http://api-intern/version from a web Pod. Clean up
afterwards.
Hint: kubectl expose takes the selector
from a Deployment; --port and --target-port
may differ.
Solution
kubectl expose deploy api --name api-intern --port 80 --target-port 8080
kubectl get svc api-intern
kubectl get endpointslices -l kubernetes.io/service-name=api-intern
kubectl exec deploy/web -- wget -qO- http://api-intern/version
kubectl get svc api-intern -o jsonpath='{.spec.selector}'; echo
kubectl delete svc api-internExpected (abridged):
service/api-intern exposed
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api-intern ClusterIP 10.96.82.30 <none> 80/TCP 1s
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
api-intern-rbvmv IPv4 8080 10.244.1.10,10.244.1.8,10.244.2.13 0s
0.2.0
{"app.kubernetes.io/name":"api"}
service "api-intern" deleted from notes namespace
The Service listens on 80, the EndpointSlice lists the container ports 8080. Two Services with the same selector are allowed; that way you can change ports or names without touching the Pods.
Save your progress? Sign up for free
Enterprise Note
- NodePorts rarely face the outside directly. In front of them sits a load balancer (cloud or MetalLB) or a Gateway that understands HTTP (Day 11, “Gateway API hands-on”).
- Because Services distribute connections, not requests, with keep-alive, HTTP/2 and gRPC all requests from one client land on one Pod. Remedy: an L7 proxy or a service mesh.
- Who may reach which Service in the cluster is governed by NetworkPolicies (Day 11). Without them, every Pod talks to every other Pod.
ndots:5makes short external names run through all search domains first. With many external calls, the fully qualified name with a trailing dot (api.example.com.) or a customizeddnsConfigpays off.