Skip to content
Hands-on IT IT Drills

Day 7: Services and Endpoints

Goal: You follow a request from the name through Service and EndpointSlice to the port in the Pod and see at which station it gets stuck. · Duration: approx. 60 min · Prerequisite: Day 6

Briefing

  • A Service is just a selector plus a virtual IP. Which Pods sit behind it is written into the EndpointSlice by the EndpointSlice controller.
  • kube-proxy translates Service IP and port into iptables rules on every node.
  • Could not connect to server only tells you that it fails. Which station is to blame is shown by the PORTS and ENDPOINTS columns of the EndpointSlice.
  • Kubernetes writes no event for any of these faults.
Fault Symptom Where the answer is
Selector does not match ENDPOINTS <unset> Selector vs. Pod labels
Wrong targetPort Endpoints present, wrong port EndpointSlice PORTS, ss
Port name missing in Pod PORTS <unset> containerPorts[].name
No load balancing always the same Pod sessionAffinity
client ──> backend (DNS) ──> 10.43.x.x:80 (Service) ──> EndpointSlice
             Day 8              kube-proxy                 Pod-IP:targetPort ──> ss

Drill 1: The path of a request

First the healthy state: two whoami Pods behind the Service backend and a tool Pod client with netshoot.

# on cp1
mkdir -p ~/debug/tag07 && cd ~/debug/tag07
kubectl create namespace tag07
cat > backend.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: backend, namespace: tag07}
spec:
  replicas: 2
  selector: {matchLabels: {app: backend}}
  template:
    metadata: {labels: {app: backend}}
    spec:
      containers:
      - name: whoami
        image: traefik/whoami:v1.11
        ports: [{name: http, containerPort: 80}]
---
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: tag07}
spec:
  selector: {app: backend}
  ports: [{name: http, port: 80, targetPort: http}]
---
apiVersion: v1
kind: Pod
metadata: {name: client, namespace: tag07, labels: {app: client}}
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot:v0.15
    command: [sleep, infinity]
EOF
kubectl apply -f backend.yaml
kubectl -n tag07 rollout status deploy/backend
kubectl -n tag07 wait pod/client --for=condition=Ready

Now walk the stations one by one – name, Service, EndpointSlice, Pod:

# on cp1
kubectl -n tag07 exec client -- dig +short backend.tag07.svc.cluster.local
kubectl -n tag07 get svc backend -o jsonpath='{.spec.clusterIP} {.spec.ports}{"\n"}'
J='{range .items[*].endpoints[*]}{.addresses[0]} {.conditions.ready} '
J+='{.targetRef.name} {.nodeName}{"\n"}{end}'
kubectl -n tag07 get endpointslice -l kubernetes.io/service-name=backend \
  -o jsonpath="$J"

Expected:

10.43.139.59
10.43.139.59 [{"name":"http","port":80,"protocol":"TCP","targetPort":"http"}]
10.42.1.29 true backend-6ddd778465-2tbqs worker1
10.42.2.26 true backend-6ddd778465-884qb worker2

And the request itself, six times through the Service, once directly to a Pod IP from your output:

# on cp1
for i in $(seq 6); do kubectl -n tag07 exec client -- curl -s backend | head -1; done
kubectl -n tag07 exec client -- curl -s 10.42.1.29 | head -1

Expected:

Hostname: backend-6ddd778465-884qb
Hostname: backend-6ddd778465-2tbqs
Hostname: backend-6ddd778465-2tbqs
Hostname: backend-6ddd778465-884qb
Hostname: backend-6ddd778465-2tbqs
Hostname: backend-6ddd778465-884qb
Hostname: backend-6ddd778465-2tbqs

Why? The first line of whoami names the Pod that answered. That way you see without any logs whether the Service balances and which Pod sits behind an IP.

Tip: kubectl get endpoints still works, but has warned since 1.33: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice. Every EndpointSlice carries the label kubernetes.io/service-name.

Save your progress? Sign up for free

Drill 2: The selector does not match

Break it:

# on cp1
cat > svc-selektor.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: tag07}
spec:
  selector: {app: backend-api}
  ports: [{name: http, port: 80, targetPort: http}]
EOF
kubectl apply -f svc-selektor.yaml

Symptom:

# on cp1
kubectl -n tag07 exec client -- curl -sS -m 5 backend

Expected:

curl: (7) Failed to connect to backend port 80 after 3 ms: Could not connect to server
command terminated with exit code 7

The name resolves (otherwise you would get Could not resolve host), the connection fails within 3 ms. Nobody is waiting behind the Service IP.

Investigate:

# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend
kubectl -n tag07 get svc backend -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n tag07 get pods --show-labels
kubectl -n tag07 events --for service/backend

Expected:

NAME            ADDRESSTYPE   PORTS     ENDPOINTS   AGE
backend-zdtk5   IPv4          <unset>   <unset>     68s
{"app":"backend-api"}
NAME                       READY   STATUS    RESTARTS   AGE   LABELS
backend-6ddd778465-2tbqs   1/1     Running   0          68s   app=backend,pod-templa...
backend-6ddd778465-884qb   1/1     Running   0          68s   app=backend,pod-templa...
client                     1/1     Running   0          68s   app=client
No events found in tag07 namespace.

Cause: The Service looks for app=backend-api, the Pods carry app=backend. The EndpointSlice stays empty, and Kubernetes reports this nowhere – no event, no status. The cross-check kubectl -n tag07 get pods -l app=backend-api returns No resources found.

Fix:

# on cp1
kubectl apply -f backend.yaml

Verify:

# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend
kubectl -n tag07 exec client -- curl -s backend | head -1

Expected:

NAME            ADDRESSTYPE   PORTS   ENDPOINTS               AGE
backend-zdtk5   IPv4          80      10.42.1.29,10.42.2.26   80s
Hostname: backend-6ddd778465-2tbqs

Why? If the readiness probe fails instead, the EndpointSlice is not empty: the IPs are in it, but with ready: false (Day 5), and get pods shows 0/1. <unset> together with 1/1 points to the selector.

Save your progress? Sign up for free

Drill 3: Endpoints present, connection still refused

Break it:

# on cp1
cat > svc-port.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: tag07}
spec:
  selector: {app: backend}
  ports: [{name: http, port: 80, targetPort: 8080}]
EOF
kubectl apply -f svc-port.yaml

Symptom:

# on cp1
kubectl -n tag07 exec client -- curl -sS -m 5 backend

Expected:

curl: (7) Failed to connect to backend port 80 after 6 ms: Could not connect to server
command terminated with exit code 7

The same message as in Drill 2 – curl alone does not tell you the station.

Investigate:

# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend
IP=$(kubectl -n tag07 get pod -l app=backend -o jsonpath='{.items[0].status.podIP}')
kubectl -n tag07 exec client -- curl -s -o /dev/null -w '%{http_code}\n' $IP:80
kubectl -n tag07 exec client -- curl -sS -m 5 $IP:8080

Expected (line wrapped):

NAME            ADDRESSTYPE   PORTS   ENDPOINTS               AGE
backend-zdtk5   IPv4          8080    10.42.1.29,10.42.2.26   82s
200
curl: (7) Failed to connect to 10.42.1.29 port 8080 after 1 ms: Could not connect to
  server
command terminated with exit code 7

Cause: The PORTS column of the EndpointSlice shows the port in the Pod (targetPort), not the Service port. Nobody listens there; the Pod answers with a TCP reset. Directly on port 80 it answers with 200.

Fix:

# on cp1
kubectl apply -f backend.yaml

Verify:

# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend
kubectl -n tag07 exec client -- curl -s backend | head -1

Expected:

NAME            ADDRESSTYPE   PORTS   ENDPOINTS               AGE
backend-zdtk5   IPv4          80      10.42.1.29,10.42.2.26   116s
Hostname: backend-6ddd778465-2tbqs

Why? targetPort: http refers to the named port in the Pod template. The number then lives in one place only, and a port change in the image only affects the Deployment.

Save your progress? Sign up for free

Drill 4: Which port does the Pod really listen on?

The number in containerPort is just documentation – it opens no port; only its name has an effect, as the target of targetPort (Drill 3). What counts is the socket in the container. whoami has no shell:

# on cp1
P=$(kubectl -n tag07 get pod -l app=backend -o jsonpath='{.items[0].metadata.name}')
kubectl -n tag07 exec $P -- sh 2>&1 | grep -o 'exec: "sh".*'

Expected:

exec: "sh": executable file not found in $PATH

So use an ephemeral container as on Day 6. It shares the Pod’s network and sees the same sockets:

# on cp1
kubectl -n tag07 debug $P --image=nicolaka/netshoot:v0.15 --target=whoami \
  --container=ports -- ss -ltnp
sleep 3
kubectl -n tag07 logs $P -c ports
kubectl -n tag07 get pod $P -o jsonpath='{.spec.containers[0].ports}{"\n"}'

Expected:

Targeting container "whoami". If you don't see processes from this container it may
be because the container runtime doesn't support this feature.
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      4096               *:80              *:*    users:(("whoami",pid=1,fd=3))
[{"containerPort":80,"name":"http","protocol":"TCP"}]

Why? *:80 means: on all addresses of the Pod. If a process listens only on 127.0.0.1:80, it is reachable via kubectl exec … curl localhost, but not through the Service – and ss shows that right away too.

Save your progress? Sign up for free

Drill 5: Endpoints without a port

Break it:

# on cp1
cat > svc-name.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: tag07}
spec:
  selector: {app: backend}
  ports: [{name: http, port: 80, targetPort: web}]
EOF
kubectl apply -f svc-name.yaml

Symptom:

# on cp1
kubectl -n tag07 exec client -- curl -sS -m 5 backend
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend

Expected:

curl: (7) Failed to connect to backend port 80 after 7 ms: Could not connect to server
command terminated with exit code 7
NAME            ADDRESSTYPE   PORTS     ENDPOINTS               AGE
backend-zdtk5   IPv4          <unset>   10.42.1.29,10.42.2.26   2m1s

Pods found, but no port: a third variant next to Drill 2 (<unset> in both columns) and Drill 3 (wrong number).

Investigate:

# on cp1
kubectl -n tag07 get svc backend -o jsonpath='{.spec.ports[0].targetPort}{"\n"}'
kubectl -n tag07 get pods -l app=backend \
  -o custom-columns=NAME:.metadata.name,PORTNAME:.spec.containers[*].ports[*].name

Expected:

web
NAME                       PORTNAME
backend-6ddd778465-2tbqs   http
backend-6ddd778465-884qb   http

Cause: targetPort: web looks in every Pod for a container port named web. There is none, so the controller enters the Pods without a port, and kube-proxy treats the Service like one without endpoints.

Fix:

# on cp1
kubectl apply -f backend.yaml

Verify:

# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend

Expected:

NAME            ADDRESSTYPE   PORTS   ENDPOINTS               AGE
backend-zdtk5   IPv4          80      10.42.2.26,10.42.1.29   2m30s

Save your progress? Sign up for free

Drill 6: What kube-proxy makes of it

In RKE2, kube-proxy runs as a static Pod on every node and writes iptables rules. You only read them. The client Pod runs on cp1, so that node’s rules apply:

# on cp1
kubectl -n kube-system get pods -l component=kube-proxy -o wide | cut -c1-79
sudo iptables-save | grep -o '"tag07/backend[^"]*"' | sort | uniq -c
sudo iptables-save | grep 'tag07/backend' | grep -o 'probability [0-9.]*'

Expected:

NAME                 READY   STATUS    RESTARTS   AGE   IP             NODE
kube-proxy-cp1       1/1     Running   0          38m   172.16.0.126   cp1
kube-proxy-worker1   1/1     Running   0          37m   172.16.0.228   worker1
kube-proxy-worker2   1/1     Running   0          37m   172.16.0.83    worker2
      1 "tag07/backend:http -> 10.42.1.29:80"
      1 "tag07/backend:http -> 10.42.2.26:80"
      2 "tag07/backend:http cluster IP"
      4 "tag07/backend:http"
probability 0.50000000000

One rule per endpoint with DNAT to Pod-IP:Port, the first with probability 0.5 – that is the entire load balancing. And without endpoints?

# on cp1
kubectl -n tag07 scale deploy backend --replicas=0
sleep 4
sudo iptables-save | grep -o '"tag07/backend[^"]*"' | sort | uniq -c
kubectl -n tag07 scale deploy backend --replicas=2
kubectl -n tag07 rollout status deploy/backend

Expected:

deployment.apps/backend scaled
      1 "tag07/backend:http has no endpoints"
deployment.apps/backend scaled
...
deployment "backend" successfully rolled out

Why? For a Service without (port-capable) endpoints, kube-proxy sets a rule REJECT --reject-with icmp-port-unreachable. That is why Drills 2 and 5 failed within a few milliseconds instead of timing out. If, on the other hand, the KUBE-SVC rules are missing entirely although the EndpointSlice is filled, the problem lies with kube-proxy itself: kubectl -n kube-system logs kube-proxy-cp1.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the Pod IPs behind the Service backend. Target time: 15 s
  2. Show selector, targetPort and endpoints of the Service in three lines. Target time: 20 s
  3. Call the Service six times and count how often each Pod answers. Target time: 30 s
  4. Call a backend Pod directly, bypassing the Service. Target time: 30 s

Solution

# on cp1
kubectl -n tag07 get endpointslice -l kubernetes.io/service-name=backend
kubectl -n tag07 describe svc backend | grep -E '^(Selector|TargetPort|Endpoints)'
for i in $(seq 6); do kubectl -n tag07 exec client -- curl -s backend | head -1
done | sort | uniq -c
kubectl -n tag07 exec client -- curl -s \
  $(kubectl -n tag07 get pod -l app=backend -o jsonpath='{.items[0].status.podIP}') \
  | head -1

Expected:

NAME            ADDRESSTYPE   PORTS   ENDPOINTS               AGE
backend-zdtk5   IPv4          80      10.42.1.31,10.42.2.28   3m47s
Selector:                 app=backend
TargetPort:               http/TCP
Endpoints:                10.42.1.31:80,10.42.2.28:80
      3 Hostname: backend-6ddd778465-5nrkz
      3 Hostname: backend-6ddd778465-pg2z9
Hostname: backend-6ddd778465-5nrkz

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag07 get endpointslice -l kubernetes.io/service-name=backend \
  -o custom-columns=NAME:.metadata.name,PORT:.ports[0].port,\
IPS:.endpoints[*].addresses[0]

Expected:

NAME            PORT   IPS
backend-zdtk5   80     10.42.1.31,10.42.2.28
# on cp1
kubectl -n tag07 exec client -- curl -s -m 5 backend | head -1

Expected:

Hostname: backend-6ddd778465-5nrkz

Challenge

Create a second Service sticky that also points to app=backend but sets sessionAffinity: ClientIP. A colleague reports: “The Service doesn’t balance, the same Pod always answers.” Prove it, find the cause in the Service and in iptables, and restore load balancing.

Hint: This time the EndpointSlice is perfectly fine.

Solution

# on cp1
cat > sticky.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: sticky, namespace: tag07}
spec:
  selector: {app: backend}
  sessionAffinity: ClientIP
  ports: [{name: http, port: 80, targetPort: http}]
EOF
kubectl apply -f sticky.yaml
sleep 3
for i in $(seq 6); do kubectl -n tag07 exec client -- curl -s sticky | head -1
done | sort | uniq -c
kubectl -n tag07 get svc sticky \
  -o jsonpath='{.spec.sessionAffinity} {.spec.sessionAffinityConfig}{"\n"}'
sudo iptables-save | grep 'tag07/sticky' | grep -o 'recent.*seconds [0-9]*' | head -1
kubectl -n tag07 patch svc sticky -p '{"spec":{"sessionAffinity":"None"}}'
for i in $(seq 6); do kubectl -n tag07 exec client -- curl -s sticky | head -1
done | sort | uniq -c

Expected:

service/sticky created
      6 Hostname: backend-6ddd778465-pg2z9
ClientIP {"clientIP":{"timeoutSeconds":10800}}
recent --rcheck --seconds 10800
service/sticky patched
      2 Hostname: backend-6ddd778465-5nrkz
      4 Hostname: backend-6ddd778465-pg2z9

With ClientIP, kube-proxy uses the iptables module recent to remember for three hours which Pod is assigned to a source IP. A single client – or an ingress controller that sends everything from one IP – then always lands on the same Pod. Random is random: 2:4 is a healthy distribution too.

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag07
sudo iptables-save | grep -c tag07

Expected:

namespace "tag07" deleted
0

Interview Note

  • A Service does not answer – first question? Does it have endpoints? get endpointslice -l kubernetes.io/service-name=…: ENDPOINTS <unset> → selector, IPs with ready: false → readiness, PORTS wrong or <unset> → targetPort.
  • What does kube-proxy do? On every node it programs DNAT rules from Service IP:port to Pod IP:targetPort; balancing is random, a Service without endpoints → REJECT.
  • Why named ports? The port number lives only in the Pod template; the Service refers to it by name.

Enterprise Note

  • Large clusters often use nftables instead of iptables, or a CNI without kube-proxy (Cilium); IPVS has been deprecated since 1.35. The EndpointSlice remains the same source of truth, you just read the rules differently.
  • Selectors on app alone are risky: two teams with app: api in the same namespace suddenly share a Service. Combine several labels.
  • sessionAffinity is rarely the right answer to session problems; stateless Pods and a shared session store scale better.
  • Service mesh faults (sidecar, mTLS) look like Drill 3 – see “Linkerd hands-on”, Day 16.