Day 7: Services and Endpoints
Goal: You follow a request from the name through Service and EndpointSlice to the port in the Pod and see at which station it gets stuck. · Duration: approx. 60 min · Prerequisite: Day 6
Briefing
- A Service is just a selector plus a virtual IP. Which Pods sit behind it is written into the EndpointSlice by the EndpointSlice controller.
- kube-proxy translates Service IP and port into iptables rules on every node.
Could not connect to serveronly tells you that it fails. Which station is to blame is shown by thePORTSandENDPOINTScolumns of the EndpointSlice.- Kubernetes writes no event for any of these faults.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Selector does not match | ENDPOINTS <unset> |
Selector vs. Pod labels |
Wrong targetPort |
Endpoints present, wrong port | EndpointSlice PORTS, ss |
| Port name missing in Pod | PORTS <unset> |
containerPorts[].name |
| No load balancing | always the same Pod | sessionAffinity |
client ──> backend (DNS) ──> 10.43.x.x:80 (Service) ──> EndpointSlice
Day 8 kube-proxy Pod-IP:targetPort ──> ss
Drill 1: The path of a request
First the healthy state: two whoami Pods behind the
Service backend and a tool Pod client with
netshoot.
# on cp1
mkdir -p ~/debug/tag07 && cd ~/debug/tag07
kubectl create namespace tag07
cat > backend.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: backend, namespace: tag07}
spec:
replicas: 2
selector: {matchLabels: {app: backend}}
template:
metadata: {labels: {app: backend}}
spec:
containers:
- name: whoami
image: traefik/whoami:v1.11
ports: [{name: http, containerPort: 80}]
---
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: tag07}
spec:
selector: {app: backend}
ports: [{name: http, port: 80, targetPort: http}]
---
apiVersion: v1
kind: Pod
metadata: {name: client, namespace: tag07, labels: {app: client}}
spec:
containers:
- name: netshoot
image: nicolaka/netshoot:v0.15
command: [sleep, infinity]
EOF
kubectl apply -f backend.yaml
kubectl -n tag07 rollout status deploy/backend
kubectl -n tag07 wait pod/client --for=condition=ReadyNow walk the stations one by one – name, Service, EndpointSlice, Pod:
# on cp1
kubectl -n tag07 exec client -- dig +short backend.tag07.svc.cluster.local
kubectl -n tag07 get svc backend -o jsonpath='{.spec.clusterIP} {.spec.ports}{"\n"}'
J='{range .items[*].endpoints[*]}{.addresses[0]} {.conditions.ready} '
J+='{.targetRef.name} {.nodeName}{"\n"}{end}'
kubectl -n tag07 get endpointslice -l kubernetes.io/service-name=backend \
-o jsonpath="$J"Expected:
10.43.139.59
10.43.139.59 [{"name":"http","port":80,"protocol":"TCP","targetPort":"http"}]
10.42.1.29 true backend-6ddd778465-2tbqs worker1
10.42.2.26 true backend-6ddd778465-884qb worker2
And the request itself, six times through the Service, once directly to a Pod IP from your output:
# on cp1
for i in $(seq 6); do kubectl -n tag07 exec client -- curl -s backend | head -1; done
kubectl -n tag07 exec client -- curl -s 10.42.1.29 | head -1Expected:
Hostname: backend-6ddd778465-884qb
Hostname: backend-6ddd778465-2tbqs
Hostname: backend-6ddd778465-2tbqs
Hostname: backend-6ddd778465-884qb
Hostname: backend-6ddd778465-2tbqs
Hostname: backend-6ddd778465-884qb
Hostname: backend-6ddd778465-2tbqs
Why? The first line of whoami names the Pod that
answered. That way you see without any logs whether the Service balances
and which Pod sits behind an IP.
Tip:
kubectl get endpointsstill works, but has warned since 1.33:v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice. Every EndpointSlice carries the labelkubernetes.io/service-name.
Save your progress? Sign up for free
Drill 2: The selector does not match
Break it:
# on cp1
cat > svc-selektor.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: tag07}
spec:
selector: {app: backend-api}
ports: [{name: http, port: 80, targetPort: http}]
EOF
kubectl apply -f svc-selektor.yamlSymptom:
# on cp1
kubectl -n tag07 exec client -- curl -sS -m 5 backendExpected:
curl: (7) Failed to connect to backend port 80 after 3 ms: Could not connect to server
command terminated with exit code 7
The name resolves (otherwise you would get
Could not resolve host), the connection fails within 3 ms.
Nobody is waiting behind the Service IP.
Investigate:
# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend
kubectl -n tag07 get svc backend -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n tag07 get pods --show-labels
kubectl -n tag07 events --for service/backendExpected:
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zdtk5 IPv4 <unset> <unset> 68s
{"app":"backend-api"}
NAME READY STATUS RESTARTS AGE LABELS
backend-6ddd778465-2tbqs 1/1 Running 0 68s app=backend,pod-templa...
backend-6ddd778465-884qb 1/1 Running 0 68s app=backend,pod-templa...
client 1/1 Running 0 68s app=client
No events found in tag07 namespace.
Cause: The Service looks for
app=backend-api, the Pods carry app=backend.
The EndpointSlice stays empty, and Kubernetes reports this nowhere – no
event, no status. The cross-check
kubectl -n tag07 get pods -l app=backend-api returns
No resources found.
Fix:
# on cp1
kubectl apply -f backend.yamlVerify:
# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend
kubectl -n tag07 exec client -- curl -s backend | head -1Expected:
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zdtk5 IPv4 80 10.42.1.29,10.42.2.26 80s
Hostname: backend-6ddd778465-2tbqs
Why? If the readiness probe fails instead, the EndpointSlice is not
empty: the IPs are in it, but with ready: false (Day 5),
and get pods shows 0/1.
<unset> together with 1/1 points to the
selector.
Save your progress? Sign up for free
Drill 3: Endpoints present, connection still refused
Break it:
# on cp1
cat > svc-port.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: tag07}
spec:
selector: {app: backend}
ports: [{name: http, port: 80, targetPort: 8080}]
EOF
kubectl apply -f svc-port.yamlSymptom:
# on cp1
kubectl -n tag07 exec client -- curl -sS -m 5 backendExpected:
curl: (7) Failed to connect to backend port 80 after 6 ms: Could not connect to server
command terminated with exit code 7
The same message as in Drill 2 – curl alone does not
tell you the station.
Investigate:
# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend
IP=$(kubectl -n tag07 get pod -l app=backend -o jsonpath='{.items[0].status.podIP}')
kubectl -n tag07 exec client -- curl -s -o /dev/null -w '%{http_code}\n' $IP:80
kubectl -n tag07 exec client -- curl -sS -m 5 $IP:8080Expected (line wrapped):
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zdtk5 IPv4 8080 10.42.1.29,10.42.2.26 82s
200
curl: (7) Failed to connect to 10.42.1.29 port 8080 after 1 ms: Could not connect to
server
command terminated with exit code 7
Cause: The PORTS column of the
EndpointSlice shows the port in the Pod
(targetPort), not the Service port. Nobody listens there;
the Pod answers with a TCP reset. Directly on port 80 it answers with
200.
Fix:
# on cp1
kubectl apply -f backend.yamlVerify:
# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backend
kubectl -n tag07 exec client -- curl -s backend | head -1Expected:
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zdtk5 IPv4 80 10.42.1.29,10.42.2.26 116s
Hostname: backend-6ddd778465-2tbqs
Why? targetPort: http refers to the
named port in the Pod template. The number then lives
in one place only, and a port change in the image only affects the
Deployment.
Save your progress? Sign up for free
Drill 4: Which port does the Pod really listen on?
The number in containerPort is just documentation – it
opens no port; only its name has an effect, as the target of
targetPort (Drill 3). What counts is the socket in the
container. whoami has no shell:
# on cp1
P=$(kubectl -n tag07 get pod -l app=backend -o jsonpath='{.items[0].metadata.name}')
kubectl -n tag07 exec $P -- sh 2>&1 | grep -o 'exec: "sh".*'Expected:
exec: "sh": executable file not found in $PATH
So use an ephemeral container as on Day 6. It shares the Pod’s network and sees the same sockets:
# on cp1
kubectl -n tag07 debug $P --image=nicolaka/netshoot:v0.15 --target=whoami \
--container=ports -- ss -ltnp
sleep 3
kubectl -n tag07 logs $P -c ports
kubectl -n tag07 get pod $P -o jsonpath='{.spec.containers[0].ports}{"\n"}'Expected:
Targeting container "whoami". If you don't see processes from this container it may
be because the container runtime doesn't support this feature.
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 4096 *:80 *:* users:(("whoami",pid=1,fd=3))
[{"containerPort":80,"name":"http","protocol":"TCP"}]
Why? *:80 means: on all addresses of the Pod. If a
process listens only on 127.0.0.1:80, it is reachable via
kubectl exec … curl localhost, but not through the Service
– and ss shows that right away too.
Save your progress? Sign up for free
Drill 5: Endpoints without a port
Break it:
# on cp1
cat > svc-name.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: tag07}
spec:
selector: {app: backend}
ports: [{name: http, port: 80, targetPort: web}]
EOF
kubectl apply -f svc-name.yamlSymptom:
# on cp1
kubectl -n tag07 exec client -- curl -sS -m 5 backend
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backendExpected:
curl: (7) Failed to connect to backend port 80 after 7 ms: Could not connect to server
command terminated with exit code 7
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zdtk5 IPv4 <unset> 10.42.1.29,10.42.2.26 2m1s
Pods found, but no port: a third variant next to Drill 2
(<unset> in both columns) and Drill 3 (wrong
number).
Investigate:
# on cp1
kubectl -n tag07 get svc backend -o jsonpath='{.spec.ports[0].targetPort}{"\n"}'
kubectl -n tag07 get pods -l app=backend \
-o custom-columns=NAME:.metadata.name,PORTNAME:.spec.containers[*].ports[*].nameExpected:
web
NAME PORTNAME
backend-6ddd778465-2tbqs http
backend-6ddd778465-884qb http
Cause: targetPort: web looks in every
Pod for a container port named web. There is none, so the
controller enters the Pods without a port, and kube-proxy treats the
Service like one without endpoints.
Fix:
# on cp1
kubectl apply -f backend.yamlVerify:
# on cp1
kubectl -n tag07 get endpointslices -l kubernetes.io/service-name=backendExpected:
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zdtk5 IPv4 80 10.42.2.26,10.42.1.29 2m30s
Save your progress? Sign up for free
Drill 6: What kube-proxy makes of it
In RKE2, kube-proxy runs as a static Pod on every node and writes
iptables rules. You only read them. The client Pod runs on
cp1, so that node’s rules apply:
# on cp1
kubectl -n kube-system get pods -l component=kube-proxy -o wide | cut -c1-79
sudo iptables-save | grep -o '"tag07/backend[^"]*"' | sort | uniq -c
sudo iptables-save | grep 'tag07/backend' | grep -o 'probability [0-9.]*'Expected:
NAME READY STATUS RESTARTS AGE IP NODE
kube-proxy-cp1 1/1 Running 0 38m 172.16.0.126 cp1
kube-proxy-worker1 1/1 Running 0 37m 172.16.0.228 worker1
kube-proxy-worker2 1/1 Running 0 37m 172.16.0.83 worker2
1 "tag07/backend:http -> 10.42.1.29:80"
1 "tag07/backend:http -> 10.42.2.26:80"
2 "tag07/backend:http cluster IP"
4 "tag07/backend:http"
probability 0.50000000000
One rule per endpoint with DNAT to Pod-IP:Port, the
first with probability 0.5 – that is the entire load balancing. And
without endpoints?
# on cp1
kubectl -n tag07 scale deploy backend --replicas=0
sleep 4
sudo iptables-save | grep -o '"tag07/backend[^"]*"' | sort | uniq -c
kubectl -n tag07 scale deploy backend --replicas=2
kubectl -n tag07 rollout status deploy/backendExpected:
deployment.apps/backend scaled
1 "tag07/backend:http has no endpoints"
deployment.apps/backend scaled
...
deployment "backend" successfully rolled out
Why? For a Service without (port-capable) endpoints, kube-proxy sets
a rule REJECT --reject-with icmp-port-unreachable. That is
why Drills 2 and 5 failed within a few milliseconds instead of timing
out. If, on the other hand, the KUBE-SVC rules are missing
entirely although the EndpointSlice is filled, the problem lies with
kube-proxy itself:
kubectl -n kube-system logs kube-proxy-cp1.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the Pod IPs behind the Service
backend. Target time: 15 s - Show selector,
targetPortand endpoints of the Service in three lines. Target time: 20 s - Call the Service six times and count how often each Pod answers. Target time: 30 s
- Call a backend Pod directly, bypassing the Service. Target time: 30 s
Solution
# on cp1
kubectl -n tag07 get endpointslice -l kubernetes.io/service-name=backend
kubectl -n tag07 describe svc backend | grep -E '^(Selector|TargetPort|Endpoints)'
for i in $(seq 6); do kubectl -n tag07 exec client -- curl -s backend | head -1
done | sort | uniq -c
kubectl -n tag07 exec client -- curl -s \
$(kubectl -n tag07 get pod -l app=backend -o jsonpath='{.items[0].status.podIP}') \
| head -1Expected:
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zdtk5 IPv4 80 10.42.1.31,10.42.2.28 3m47s
Selector: app=backend
TargetPort: http/TCP
Endpoints: 10.42.1.31:80,10.42.2.28:80
3 Hostname: backend-6ddd778465-5nrkz
3 Hostname: backend-6ddd778465-pg2z9
Hostname: backend-6ddd778465-5nrkz
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag07 get endpointslice -l kubernetes.io/service-name=backend \
-o custom-columns=NAME:.metadata.name,PORT:.ports[0].port,\
IPS:.endpoints[*].addresses[0]Expected:
NAME PORT IPS
backend-zdtk5 80 10.42.1.31,10.42.2.28
# on cp1
kubectl -n tag07 exec client -- curl -s -m 5 backend | head -1Expected:
Hostname: backend-6ddd778465-5nrkz
Challenge
Create a second Service sticky that also points to
app=backend but sets
sessionAffinity: ClientIP. A colleague reports: “The
Service doesn’t balance, the same Pod always answers.” Prove it, find
the cause in the Service and in iptables, and restore load
balancing.
Hint: This time the EndpointSlice is perfectly fine.
Solution
# on cp1
cat > sticky.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: sticky, namespace: tag07}
spec:
selector: {app: backend}
sessionAffinity: ClientIP
ports: [{name: http, port: 80, targetPort: http}]
EOF
kubectl apply -f sticky.yaml
sleep 3
for i in $(seq 6); do kubectl -n tag07 exec client -- curl -s sticky | head -1
done | sort | uniq -c
kubectl -n tag07 get svc sticky \
-o jsonpath='{.spec.sessionAffinity} {.spec.sessionAffinityConfig}{"\n"}'
sudo iptables-save | grep 'tag07/sticky' | grep -o 'recent.*seconds [0-9]*' | head -1
kubectl -n tag07 patch svc sticky -p '{"spec":{"sessionAffinity":"None"}}'
for i in $(seq 6); do kubectl -n tag07 exec client -- curl -s sticky | head -1
done | sort | uniq -cExpected:
service/sticky created
6 Hostname: backend-6ddd778465-pg2z9
ClientIP {"clientIP":{"timeoutSeconds":10800}}
recent --rcheck --seconds 10800
service/sticky patched
2 Hostname: backend-6ddd778465-5nrkz
4 Hostname: backend-6ddd778465-pg2z9
With ClientIP, kube-proxy uses the iptables module
recent to remember for three hours which Pod is assigned to
a source IP. A single client – or an ingress controller that sends
everything from one IP – then always lands on the same Pod. Random is
random: 2:4 is a healthy distribution too.
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag07
sudo iptables-save | grep -c tag07Expected:
namespace "tag07" deleted
0
Interview Note
- A Service does not answer – first question? Does it have
endpoints?
get endpointslice -l kubernetes.io/service-name=…:ENDPOINTS <unset>→ selector, IPs withready: false→ readiness,PORTSwrong or<unset>→targetPort. - What does kube-proxy do? On every node it programs DNAT
rules from Service IP:port to Pod IP:targetPort; balancing is random, a
Service without endpoints →
REJECT. - Why named ports? The port number lives only in the Pod template; the Service refers to it by name.
Enterprise Note
- Large clusters often use
nftablesinstead of iptables, or a CNI without kube-proxy (Cilium); IPVS has been deprecated since 1.35. The EndpointSlice remains the same source of truth, you just read the rules differently. - Selectors on
appalone are risky: two teams withapp: apiin the same namespace suddenly share a Service. Combine several labels. sessionAffinityis rarely the right answer to session problems; stateless Pods and a shared session store scale better.- Service mesh faults (sidecar, mTLS) look like Drill 3 – see “Linkerd hands-on”, Day 16.