Skip to content
Hands-on IT IT Drills

Day 16: Services and Ingress

Goal: You make notes-api reachable inside the cluster and from outside via a Service, find selector and port errors in seconds, and route HTTP and HTTPS by host and path with an Ingress. · Duration: approx. 75 min · Prerequisite: Day 14

Briefing

CKAD: Services and Networking (20 %) – “Use Kubernetes primitives to implement common networking patterns” (Services) and “Use Ingress rules to expose applications”.

 outside --> NodeIP:30080 (Traefik) --host/path--> Service notes:80 --+
 outside --> NodeIP:30616 (NodePort) ---------------------------------+--> Pods :8080
 Pod     --> notes.tag16 (CoreDNS) --> ClusterIP --> EndpointSlice ---+

A Service finds its Pods only through the selector; port is the port of the Service, targetPort the port in the container.

mkdir -p ~/ckad/tag16 && cd ~/ckad/tag16
k create ns tag16
k config set-context --current --namespace=tag16
k create secret generic notes-key --from-literal=NOTES_API_KEY=lab-key-123

lab-key-123 is a pure lab value.

Drill 1: ClusterIP with k expose

The container port gets a name (http) – the Service refers to it instead of to the number:

cat > notes.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0
        ports:
        - name: http
          containerPort: 8080
        envFrom:
        - secretRef:
            name: notes-key
        readinessProbe:
          httpGet:
            path: /healthz
            port: http
EOF
k apply -f notes.yaml
k rollout status deployment notes
k expose deployment notes --port=80 --target-port=http
k get svc notes
k get svc notes -o yaml | sed -n '/^  ports:/,/^  type:/p'

Expected (abridged):

service/notes exposed
NAME    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
notes   ClusterIP   10.97.22.56   <none>        80/TCP    1s
  ports:
  - port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: notes
  sessionAffinity: None
  type: ClusterIP

k expose takes the selector from the Deployment. A client Pod for testing:

k run client --image=busybox:1.37 -- sleep 3600
k wait --for=condition=Ready pod/client
k exec client -- wget -qO- notes/version
k exec client -- wget -qO- --header 'X-API-Key: lab-key-123' \
  --post-data '{"text":"a"}' notes/notes; echo
for i in 1 2 3 4; do
  k exec client -- wget -qO- --header 'X-API-Key: lab-key-123' notes/notes
done

Expected (order is random):

1.0
{"id":"ff7acdfb-07f3-430c-9245-1290aa0879ac","text":"a"}
[]
[{"id":"ff7acdfb-07f3-430c-9245-1290aa0879ac","text":"a"}]
[{"id":"ff7acdfb-07f3-430c-9245-1290aa0879ac","text":"a"}]
[]

Each Pod has its own in-memory store – this way you can see that the Service distributes the connections. The container port is not open on the ClusterIP:

k exec client -- wget -T 3 -qO- $(k get svc notes -o jsonpath='{.spec.clusterIP}'):8080

Expected:

wget: download timed out
command terminated with exit code 1

Save your progress? Sign up for free

Drill 2: EndpointSlices – who is behind it?

k get endpointslices -l kubernetes.io/service-name=notes
k scale deployment notes --replicas=3
k rollout status deployment notes
J='{range .items[*].endpoints[*]}{.addresses[0]} {.targetRef.name}{"\n"}{end}'
k get endpointslices -l kubernetes.io/service-name=notes -o jsonpath="$J"

Expected:

NAME          ADDRESSTYPE   PORTS   ENDPOINTS                        AGE
notes-ct7wm   IPv4          8080    192.168.189.95,192.168.235.165   2m20s
deployment.apps/notes scaled
deployment "notes" successfully rolled out
192.168.189.95 notes-7f56bf4556-kzxlc
192.168.235.165 notes-7f56bf4556-4lsnr
192.168.235.170 notes-7f56bf4556-9kwzj

k get endpoints notes and k describe svc notes show the same; Endpoints, however, reports Warning: v1 Endpoints is deprecated in v1.33+. Only Pods whose labels match and that are ready end up in the list. To take a Pod out of traffic for debugging without deleting it:

P=$(k get pods -l app=notes -o name | head -1)
k label $P app=quarantaene --overwrite
k get pods -L app
k get endpointslices -l kubernetes.io/service-name=notes -o jsonpath="$J"

Expected (abridged):

pod/notes-7f56bf4556-4lsnr labeled
NAME                     READY   STATUS    RESTARTS   AGE     APP
notes-7f56bf4556-4lsnr   1/1     Running   0          2m35s   quarantaene
notes-7f56bf4556-94wgr   1/1     Running   0          4s      notes
notes-7f56bf4556-9kwzj   1/1     Running   0          11s     notes
notes-7f56bf4556-kzxlc   1/1     Running   0          2m35s   notes
192.168.189.95 notes-7f56bf4556-kzxlc
192.168.235.170 notes-7f56bf4556-9kwzj
192.168.235.173 notes-7f56bf4556-94wgr

The ReplicaSet no longer counts the Pod and starts a replacement. Clean up afterwards: k delete pod -l app=quarantaene.

Save your progress? Sign up for free

Drill 3: DNS names and headless Service

k exec client -- cat /etc/resolv.conf
k exec client -- wget -qO- notes.tag16/version
k exec client -- wget -qO- notes.tag16.svc.cluster.local/version

Expected:

search tag16.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5
1.0
1.0

In the same namespace notes is enough, from other namespaces notes.tag16, fully qualified notes.tag16.svc.cluster.local. A headless Service (clusterIP: None) has no IP of its own – DNS returns the Pod IPs directly:

k expose deployment notes --name=notes-hl --cluster-ip=None --port=8080
k get svc notes-hl
k exec client -- nslookup notes-hl.tag16.svc.cluster.local

Expected (abridged):

service/notes-hl exposed
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
notes-hl   ClusterIP   None         <none>        8080/TCP   0s
Name:   notes-hl.tag16.svc.cluster.local
Address: 192.168.235.173
Name:   notes-hl.tag16.svc.cluster.local
Address: 192.168.235.170
Name:   notes-hl.tag16.svc.cluster.local
Address: 192.168.189.95

Headless Services need StatefulSets (Day 3) for stable names like db-0.db.tag16.svc.cluster.local. With short names, busybox nslookup tries the whole search list and reports many NXDOMAIN – the FQDN is cleaner.

Save your progress? Sign up for free

Drill 4: NodePort and LoadBalancer without a cloud

For a fixed NodePort you need YAML – k create service generates it, but sets a wrong selector:

k create service nodeport notes-np --tcp=80:http --node-port=30616 $do > notes-np.yaml
grep -A1 selector notes-np.yaml
sed -i '/selector:/{n;s/notes-np/notes/}' notes-np.yaml
k apply -f notes-np.yaml
k get svc notes-np
for ip in 172.16.0.126 172.16.0.228 172.16.0.83; do curl -s $ip:30616/version; done

Expected:

  selector:
    app: notes-np
service/notes-np created
NAME       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
notes-np   NodePort   10.102.89.223   <none>        80:30616/TCP   2s
1.0
1.0
1.0

k create service always sets app: <name> – almost never what you want. The NodePort is open on every node (range 30000–32767). Type LoadBalancer:

k expose deployment notes --name=notes-lb --type=LoadBalancer --port=80 \
  --target-port=http
k get svc notes-lb
NP=$(k get svc notes-lb -o jsonpath='{.spec.ports[0].nodePort}')
curl -s 172.16.0.228:$NP/version

Expected:

service/notes-lb exposed
NAME       TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
notes-lb   LoadBalancer   10.103.68.121   <pending>     80:30462/TCP   5s
1.0

The external IP is provided by a cloud controller or MetalLB – neither exists in the lab, so <pending> stays forever. The NodePort underneath works anyway.

Save your progress? Sign up for free

Drill 5: Finding selector and port errors

cat > notes-bad.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes-bad
spec:
  selector:
    app: note
  ports:
  - port: 80
EOF
k apply -f notes-bad.yaml
k exec client -- wget -T 3 -qO- notes-bad/version
k get endpointslices -l kubernetes.io/service-name=notes-bad
k get svc notes-bad -o wide
k get pods -l app=notes --show-labels | head -2

Expected:

wget: can't connect to remote host (10.100.200.240): Connection refused
command terminated with exit code 1
NAME              ADDRESSTYPE   PORTS     ENDPOINTS   AGE
notes-bad-9x7xr   IPv4          <unset>   <unset>     2s
NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE   SELECTOR
notes-bad   ClusterIP   10.100.200.240   <none>        80/TCP    3s    app=note
NAME                     READY   STATUS    RESTARTS   AGE   LABELS
notes-7f56bf4556-94wgr   1/1     Running   0          51s   app=notes,pod-template-...

Empty endpoints = the selector does not match. Fix it:

k set selector svc notes-bad app=notes
k get endpointslices -l kubernetes.io/service-name=notes-bad
k exec client -- wget -T 3 -qO- notes-bad/version

Expected:

service/notes-bad selector updated
NAME              ADDRESSTYPE   PORTS   ENDPOINTS
notes-bad-9x7xr   IPv4          80      192.168.235.173,192.168.189.95,192.168.235.170
wget: can't connect to remote host (10.100.200.240): Connection refused
command terminated with exit code 1

Endpoints are there, but PORTS 80: without targetPort, Kubernetes uses the value of port. The app, however, listens on 8080:

k patch svc notes-bad -p '{"spec":{"ports":[{"port":80,"targetPort":"http"}]}}'
sleep 3
k exec client -- wget -T 3 -qO- notes-bad/version

Expected:

service/notes-bad patched
1.0

kube-proxy needs a second or two after changes. Rule of thumb: empty endpoints → selector, endpoints there, but errors → targetPort (or app/readiness).

Save your progress? Sign up for free

Drill 6: Ingress with host rules

Traefik is the lab’s Ingress controller (IngressClass traefik, default), reachable via NodePort 30080 (HTTP) and 30443 (HTTPS). Add a second app:

k create deployment web --image=nginx:1.29 --port=80
k expose deployment web --port=80
k create ingress host --rule="notes.tag16.example/*=notes:80" \
  --rule="web.tag16.example/*=web:80"
k get ingress host
k get ingress host -o jsonpath='{.spec.ingressClassName}{"\n"}'

Expected:

ingress.networking.k8s.io/host created
NAME   CLASS     HOSTS                                   ADDRESS   PORTS   AGE
host   traefik   notes.tag16.example,web.tag16.example             80      4s
traefik

Without --class, the API server fills in the default IngressClass. ADDRESS stays empty because Traefik is only reachable via NodePort here. Without DNS, you test with the Host header:

curl -s -H 'Host: notes.tag16.example' 172.16.0.126:30080/version
curl -s -H 'Host: web.tag16.example' 172.16.0.126:30080/ | grep title
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: x.tag16.example' 172.16.0.126:30080/
k describe ingress host | sed -n '/Rules/,/Annotations/p'

Expected:

1.0
<title>Welcome to nginx!</title>
404
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  notes.tag16.example
                       /   notes:80 (192.168.189.95:8080,192.168.235.170:8080,...)
  web.tag16.example
                       /   web:80 (192.168.235.133:80)
Annotations:           <none>

describe shows the Pod IPs behind each backend – empty parentheses mean: a Service without endpoints or the wrong port.

Save your progress? Sign up for free

Drill 7: Paths and pathType

One host, two apps: /version exactly and /notes as a prefix go to notes-api, the rest to nginx. In --rule, a trailing * makes it Prefix, without * it becomes Exact:

H=shop.tag16.example
k create ingress pfad --class=traefik --rule="$H/version=notes:80" \
  --rule="$H/notes*=notes:80" --rule="$H/*=web:80" $do > pfad.yaml
grep -E 'path' pfad.yaml
k apply -f pfad.yaml

Expected:

      paths:
        path: /version
        pathType: Exact
        path: /notes
        pathType: Prefix
        path: /
        pathType: Prefix
ingress.networking.k8s.io/pfad created

Who answers? notes-api returns text/plain or JSON, nginx text/html:

for p in /version /version/ /notes /notesx /xyz; do
  echo "$p $(curl -s -o /dev/null -w '%{http_code} %{content_type}' -H "Host: $H" \
    172.16.0.126:30080$p)"
done

Expected:

/version 200 text/plain; charset=utf-8
/version/ 404 text/html
/notes 200 application/json
/notesx 404 text/plain; charset=utf-8
/xyz 404 text/html

Exact matches only /version – even /version/ goes to nginx. According to the specification, Prefix compares whole path elements: /notes matches /notes/x, not /notesx (k explain ingress.spec.rules.http.paths.pathType). Traefik 3.7.19 deviates from this and sends /notesx to notes-api; only the option --providers.kubernetesingress.strictprefixmatching (default false) makes it spec-compliant. In the exam, the specification counts.

Save your progress? Sign up for free

Drill 8: TLS at the Ingress

H=secure.tag16.example
openssl req -x509 -newkey rsa:2048 -nodes -days 30 -keyout tls.key -out tls.crt \
  -subj "/CN=$H" -addext "subjectAltName=DNS:$H"
k create secret tls secure-tls --cert=tls.crt --key=tls.key
k create ingress secure --class=traefik --rule="$H/*=notes:80,tls=secure-tls"
k get ingress secure
k get ingress secure -o yaml | sed -n '/^  tls:/,/^status/p'

Expected (abridged):

secret/secure-tls created
ingress.networking.k8s.io/secure created
NAME     CLASS     HOSTS                  ADDRESS   PORTS     AGE
secure   traefik   secure.tag16.example             80, 443   3s
  tls:
  - hosts:
    - secure.tag16.example
    secretName: secure-tls
status:

--resolve replaces the DNS entry and ensures the correct SNI name, --cacert trusts the self-signed certificate:

curl -s --resolve $H:30443:172.16.0.126 --cacert tls.crt https://$H:30443/version
curl -sv --resolve $H:30443:172.16.0.126 --cacert tls.crt https://$H:30443/version \
  2>&1 | grep -E 'subject:|verify ok'
curl -skv -H "Host: $H" https://172.16.0.126:30443/version 2>&1 | grep 'subject:'

Expected:

1.0
*  subject: CN=secure.tag16.example
*  SSL certificate verify ok.
*  subject: CN=TRAEFIK DEFAULT CERT

With only the Host header, the SNI name is missing – Traefik then shows its default certificate. The host remains reachable over HTTP (30080); a redirect to HTTPS is a controller matter (a middleware in Traefik), not part of the Ingress object.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Make the Deployment web reachable as NodePort Service web-np on port 8081. Target time: 30 s
  2. Start Pod cache (nginx:1.29, port 80) together with a ClusterIP Service in one command. Target time: 20 s
  3. Ingress api: host api.tag16.example, only exactly /version to notes:80. Target time: 45 s
  4. Show the Pod IPs behind the Service notes. Target time: 20 s

Solution

k expose deployment web --name=web-np --type=NodePort --port=8081 --target-port=80
k run cache --image=nginx:1.29 --port=80 --expose
k create ingress api --rule="api.tag16.example/version=notes:80"
curl -s -H 'Host: api.tag16.example' 172.16.0.126:30080/version
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: api.tag16.example' \
  172.16.0.126:30080/notes
k get endpointslices -l kubernetes.io/service-name=notes

Expected (abridged):

service/web-np exposed
pod/cache created
service/cache created
ingress.networking.k8s.io/api created
1.0
404
NAME          ADDRESSTYPE   PORTS   ENDPOINTS                                        AGE
notes-ct7wm   IPv4          8080    192.168.189.95,192.168.235.170,192.168.235.173   5m

Save your progress? Sign up for free

Checkpoint

k get svc

Expected: among others notes (ClusterIP), notes-hl (None), notes-np (80:30616/TCP), notes-lb (<pending>), web-np (NodePort).

k get svc notes-bad -o jsonpath='{.spec.selector.app} {.spec.ports[0].targetPort}{"\n"}'

Expected: notes http

curl -s -H 'Host: notes.tag16.example' 172.16.0.126:30080/version

Expected: 1.0

curl -s --resolve secure.tag16.example:30443:172.16.0.126 --cacert tls.crt \
  https://secure.tag16.example:30443/version

Expected: 1.0

k exec client -- nslookup notes-hl.tag16.svc.cluster.local

Expected: after the DNS server (10.96.0.10), three Pod IPs (192.168.…).

Clean up – only once you have also done the challenge:

k config set-context --current --namespace=default
k delete ns tag16

Expected: namespace "tag16" deleted. No later day needs anything from tag16.

Challenge

A colleague published notes-api under fix.tag16.example, but all you get is 404. Make curl -H 'Host: fix.tag16.example' 172.16.0.126:30080/version answer without touching the Deployment.

cat > fix.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: fix
spec:
  selector:
    app: notes-api
  ports:
  - name: web
    port: 80
    targetPort: 8081
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fix
spec:
  ingressClassName: nginx
  rules:
  - host: fix.tag16.example
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: fix
            port:
              name: http
EOF
k apply -f fix.yaml

Hint: There are four errors; the HTTP status (404, 503, 502) tells you where you are.

Solution

k get ingressclass
k get ingress fix
k describe ingress fix | grep 'fix:'
k get svc fix -o wide
k get endpointslices -l kubernetes.io/service-name=fix

Expected:

NAME                CONTROLLER                      PARAMETERS   AGE
traefik (default)   traefik.io/ingress-controller   <none>       60m
NAME   CLASS   HOSTS               ADDRESS   PORTS   AGE
fix    nginx   fix.tag16.example             80      4s
                     /   fix:http ()
NAME   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE   SELECTOR
fix    ClusterIP   10.97.97.245   <none>        80/TCP    4s    app=notes-api
NAME        ADDRESSTYPE   PORTS     ENDPOINTS   AGE
fix-g467m   IPv4          <unset>   <unset>     4s

The errors: the IngressClass nginx does not exist (404), the Ingress refers to the port name http while the Service calls it web (still 404), the selector matches no Pod (503), and targetPort: 8081 is wrong (502). Fix everything at once:

sed -i -e 's/ingressClassName: nginx/ingressClassName: traefik/' \
  -e 's/name: http$/name: web/' -e 's/app: notes-api/app: notes/' \
  -e 's/targetPort: 8081/targetPort: http/' fix.yaml
k apply -f fix.yaml
sleep 3
curl -s -H 'Host: fix.tag16.example' 172.16.0.126:30080/version

Expected:

service/fix configured
ingress.networking.k8s.io/fix configured
1.0

Traefik names the reason for the 404 in its own log: k logs -n traefik deploy/traefik --since=1m shows Cannot create service error="service port not found".

Save your progress? Sign up for free

Exam Tip

  • k expose takes over the selector, k create service sets app: <name> – after every Service, check k get endpointslices -l kubernetes.io/service-name=<svc>.
  • k create ingress --rule="host/pfad*=svc:port,tls=secret": * = Prefix, without * = Exact. Test with curl -H 'Host: …' against the controller’s address.
  • Bookmark the docs pages “Service”, “Ingress” and “DNS for Services and Pods”.

Enterprise Note

  • External IPs come from the cloud controller or, on bare metal, from MetalLB/Cilium; NodePorts are rarely opened to the outside.
  • The successor to Ingress is the Gateway API (GatewayClass, Gateway, HTTPRoute) with header routing, weights and role separation – not installed in the lab.
  • cert-manager (e.g. Let’s Encrypt) manages certificates instead of openssl by hand.
  • Ingress annotations are controller-specific; switching controllers (for example away from the no-longer-maintained ingress-nginx) means reviewing manifests.
  • externalTrafficPolicy: Local preserves the client IP, but only distributes to nodes with a matching Pod.