Day 9: Endpoints and Availability
Goal: You read EndpointSlices including their conditions, control with readiness,
publishNotReadyAddresses, preStop andtrafficDistributionwhich Pods get traffic, and verify it in Cilium. · Duration: approx. 70 min · Prerequisite: Day 6
Briefing
- CKNE: Service Networking and DNS (25 %) – endpoints decide whether a Service answers at all; the most common reason for “the Service does not work”.
- Every endpoint has three conditions:
ready,serving,terminating. Cilium translates them into backend states (active,maintenance,terminating). - The backend’s readiness probe (agnhost
netexec) checkstest ! -e /tmp/down– that is how you toggle readiness withtouch.
Pod Ready? ── kubelet ──> Pod status ──> EndpointSlice controller ──> EndpointSlice
│
Cilium agent (eBPF backends) <──┘
Drill 1: Read EndpointSlices
mkdir -p ~/ckne/tag09 && cd ~/ckne/tag09
k create ns tag09
k config set-context --current --namespace=tag09
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
terminationGracePeriodSeconds: 60
containers:
- name: web
image: registry.k8s.io/e2e-test-images/agnhost:2.66.1
args: ["netexec", "--http-port=8080", "--delay-shutdown=45"]
ports:
- containerPort: 8080
readinessProbe:
exec:
command: ["sh", "-c", "test ! -e /tmp/down"]
periodSeconds: 2
lifecycle:
preStop:
sleep:
seconds: 15
EOF
k apply -f web.yaml
k expose deployment web --port=80 --target-port=8080
k run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k rollout status deployment web
k get pods -o wide
k get endpointslices -l kubernetes.io/service-name=web -o yamlExpected (abridged):
deployment "web" successfully rolled out
NAME READY STATUS ... IP NODE
client 1/1 Running ... 10.0.2.186 worker2
web-565f96c64-l72c9 1/1 Running ... 10.0.1.47 worker1
web-565f96c64-n2pjp 1/1 Running ... 10.0.2.214 worker2
- addressType: IPv4
endpoints:
- addresses:
- 10.0.1.47
conditions:
ready: true
serving: true
terminating: false
nodeName: worker1
targetRef:
kind: Pod
name: web-565f96c64-l72c9
...
metadata:
labels:
endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
kubernetes.io/service-name: web
name: web-9ts49
ownerReferences:
- kind: Service
name: web
ports:
- name: ""
port: 8080
protocol: TCP
The slice belongs to the Service (ownerReferences) and
finds it via the label kubernetes.io/service-name. The port
is the container port, not the Service port. To fit the three conditions
on one line, you create a jsonpath template:
JP='{range .items[*].endpoints[*]}{.targetRef.name}{" r="}{.conditions.ready}'\
'{" s="}{.conditions.serving}{" t="}{.conditions.terminating}{"\n"}{end}'
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"Expected:
web-565f96c64-l72c9 r=true s=true t=false
web-565f96c64-n2pjp r=true s=true t=false
Save your progress? Sign up for free
Drill 2: Readiness controls the endpoints
P1=$(k get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
k exec $P1 -- touch /tmp/down
sleep 8
k get pods -l app=web
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"
k get endpointslices -l kubernetes.io/service-name=web
for i in 1 2 3 4; do k exec client -- curl -s http://web/hostname; echo; done
CIP=$(k get svc web -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$CIP:"Expected:
NAME READY STATUS RESTARTS AGE
web-565f96c64-l72c9 0/1 Running 0 28s
web-565f96c64-n2pjp 1/1 Running 0 28s
web-565f96c64-l72c9 r=false s=false t=false
web-565f96c64-n2pjp r=true s=true t=false
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
web-9ts49 IPv4 8080 10.0.1.47,10.0.2.214 29s
web-565f96c64-n2pjp
web-565f96c64-n2pjp
web-565f96c64-n2pjp
web-565f96c64-n2pjp
69 10.103.176.39:80/TCP ClusterIP 1 => 10.0.1.47:8080/TCP (maintenance)
2 => 10.0.2.214:8080/TCP (active)
After three failed probes (failureThreshold 3 ×
periodSeconds 2), the Pod is 0/1, the slice
sets ready and serving to false,
and Cilium parks the backend as maintenance. Trap: the
ENDPOINTS column of k get endpointslices shows
all addresses, including unready ones – only the
conditions tell the truth. The Pod stays like this until Drill 3 removes
the file again.
Save your progress? Sign up for free
Drill 3: publishNotReadyAddresses
Some services (database clusters, etcd) must find their peers before readiness. Two headless Services for comparison:
k expose deployment web --name=web-hl --cluster-ip=None --port=8080
k expose deployment web --name=web-peers --cluster-ip=None --port=8080 \
$do > web-peers.yaml
sed -i '/^ clusterIP: None/a\ publishNotReadyAddresses: true' web-peers.yaml
grep -A1 clusterIP web-peers.yaml
k apply -f web-peers.yaml
k exec client -- dig +short web-hl.tag09.svc.cluster.local
k exec client -- dig +short web-peers.tag09.svc.cluster.local
k get endpointslices -l kubernetes.io/service-name=web-peers -o jsonpath="$JP"Expected:
service/web-hl exposed
clusterIP: None
publishNotReadyAddresses: true
service/web-peers created
10.0.2.214
10.0.1.47
10.0.2.214
web-565f96c64-n2pjp r=true s=true t=false
web-565f96c64-l72c9 r=true s=false t=false
web-hl returns only the ready Pod,
web-peers both. In the slice this shows as
ready=true despite serving=false – the
controller ignores readiness, as
k explain service.spec.publishNotReadyAddresses describes.
Now make the Pod healthy again:
k exec $P1 -- rm /tmp/down
sleep 4
k exec client -- dig +short web-hl.tag09.svc.cluster.local
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$CIP:"Expected:
10.0.2.214
10.0.1.47
69 10.103.176.39:80/TCP ClusterIP 1 => 10.0.1.47:8080/TCP (active)
2 => 10.0.2.214:8080/TCP (active)
Save your progress? Sign up for free
Drill 4: Terminating endpoints, preStop and grace period
What happens to a Pod that is deleted while traffic is still coming in?
P1=$(k get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
PIP=$(k get pod $P1 -o jsonpath='{.status.podIP}')
k delete pod $P1 --wait=false
sleep 3
k get pods -l app=web
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A2 "$CIP:"
for i in 1 2 3 4; do k exec client -- curl -s http://web/hostname; echo; done
k exec client -- curl -s -m2 http://$PIP:8080/hostname; echo " <- direkt"Expected:
pod "web-565f96c64-l72c9" deleted from tag09 namespace
NAME READY STATUS RESTARTS AGE
web-565f96c64-dtw9w 1/1 Running 0 4s
web-565f96c64-l72c9 1/1 Terminating 0 69s
web-565f96c64-n2pjp 1/1 Running 0 69s
web-565f96c64-l72c9 r=false s=true t=true
web-565f96c64-n2pjp r=true s=true t=false
web-565f96c64-dtw9w r=true s=true t=false
69 10.103.176.39:80/TCP ClusterIP 1 => 10.0.1.47:8080/TCP (terminating)
2 => 10.0.1.120:8080/TCP (active)
3 => 10.0.2.214:8080/TCP (active)
web-565f96c64-dtw9w
web-565f96c64-n2pjp
web-565f96c64-dtw9w
web-565f96c64-n2pjp
web-565f96c64-l72c9 <- direkt
Immediately on deletion, the endpoint becomes
terminating and ready=false – new connections
only go to the other Pods. serving=true means: the Pod
could still answer, and it does, as the direct call shows. The timing:
first 15 s of preStop.sleep, then SIGTERM, which agnhost
bridges for another 45 s with --delay-shutdown=45 –
together exactly the terminationGracePeriodSeconds: 60. In
that time, Cilium agents and typical external load balancers have
usually removed the endpoint. After a good minute:
k get pods -l app=web
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"Expected:
NAME READY STATUS RESTARTS AGE
web-565f96c64-dtw9w 1/1 Running 0 73s
web-565f96c64-n2pjp 1/1 Running 0 2m18s
web-565f96c64-n2pjp r=true s=true t=false
web-565f96c64-dtw9w r=true s=true t=false
Save your progress? Sign up for free
Drill 5: trafficDistribution and zones
k explain service.spec.trafficDistribution | sed -n '/DESCRIPTION/,$p'
k patch svc web -p '{"spec":{"trafficDistribution":"PreferBogus"}}'Expected (abridged):
DESCRIPTION:
TrafficDistribution offers a way to express preferences for how traffic is
distributed to Service endpoints. ... If set
to "PreferClose", implementations should prioritize endpoints that are in
the same zone.
The Service "web" is invalid: spec.trafficDistribution: Unsupported value:
"PreferBogus": supported values: "PreferClose", "PreferSameZone", "PreferSameNode"
k explain only names PreferClose; which
values 1.37 really knows is revealed by the validation. Zones come from
the node label topology.kubernetes.io/zone – in the lab you
set them yourself:
k label node worker1 topology.kubernetes.io/zone=zone-a
k label node worker2 topology.kubernetes.io/zone=zone-b
k get nodes -L topology.kubernetes.io/zone
k patch svc web -p '{"spec":{"trafficDistribution":"PreferSameZone"}}'
sleep 3
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{range .items[*].endpoints[*]}{.nodeName} {.zone} {.hints}{"\n"}{end}'
k patch svc web -p '{"spec":{"trafficDistribution":"PreferSameNode"}}'
sleep 3
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{range .items[*].endpoints[*]}{.nodeName} {.zone} {.hints}{"\n"}{end}'Expected (abridged):
node/worker1 labeled
node/worker2 labeled
NAME STATUS ROLES AGE VERSION ZONE
cp1 Ready control-plane 53m v1.37.2
worker1 Ready <none> 47m v1.37.2 zone-a
worker2 Ready <none> 47m v1.37.2 zone-b
service/web patched
worker2 zone-b {"forZones":[{"name":"zone-b"}]}
worker1 zone-a {"forZones":[{"name":"zone-a"}]}
service/web patched
worker2 zone-b {"forNodes":[{"name":"worker2"}],"forZones":[{"name":"zone-b"}]}
worker1 zone-a {"forNodes":[{"name":"worker1"}],"forZones":[{"name":"zone-a"}]}
The controller writes hints (hints) into the slice –
nothing more. Whether they take effect is decided by the Service
implementation. The client runs on worker2 (zone b):
for i in 1 2 3 4 5 6; do k exec client -- curl -s http://web/hostname; echo; done
cilium config view | grep service-topologyExpected:
web-565f96c64-n2pjp
web-565f96c64-n2pjp
web-565f96c64-dtw9w
web-565f96c64-dtw9w
web-565f96c64-n2pjp
web-565f96c64-n2pjp
enable-service-topology false
Both zones answer: Cilium only evaluates hints with the Helm value
loadBalancer.serviceTopology=true (agent option
enable-service-topology). In the lab that stays off – in
the exam the rule is: first check whether the implementation honors
hints before you hunt for “wrong” routing. PreferClose is
the old name and now returns a warning:
k patch svc web -p '{"spec":{"trafficDistribution":"PreferClose"}}'Expected:
Warning: spec.trafficDistribution: "PreferClose" is deprecated; use "PreferSameZone"
service/web patched
Save your progress? Sign up for free
Drill 6: PodDisruptionBudget – readiness counts
A PDB protects against voluntary disruptions (drain, eviction) and only counts ready Pods – the same readiness that controls the endpoints.
k create pdb web --selector=app=web --min-available=1
k get pdb web
P1=$(k get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
P2=$(k get pod -l app=web -o jsonpath='{.items[1].metadata.name}')
k exec $P1 -- touch /tmp/down
sleep 8
k get pdb web
cat > evict.json <<EOF
{"apiVersion":"policy/v1","kind":"Eviction","metadata":{"name":"$P2"}}
EOF
k create --raw /api/v1/namespaces/tag09/pods/$P2/eviction -f evict.jsonExpected:
poddisruptionbudget.policy/web created
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
web 1 N/A 1 2s
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
web 1 N/A 0 11s
Error from server (TooManyRequests): Cannot evict pod as it would violate the pod's
disruption budget.
k drain sends exactly this eviction for every Pod –
without practicing a node drain, you see here why a drain hangs: an
unready Pod uses up the budget. Make the Pod healthy again, then it
works:
k exec $P1 -- rm /tmp/down
sleep 5
k get pdb web
k create --raw /api/v1/namespaces/tag09/pods/$P2/eviction -f evict.json; echo
k get pods -l app=webExpected:
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
web 1 N/A 1 17s
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","code":201}
NAME READY STATUS RESTARTS AGE
web-565f96c64-dtw9w 1/1 Running 0 2m10s
web-565f96c64-n2pjp 1/1 Terminating 0 3m15s
web-565f96c64-p2zj9 1/1 Running 0 3s
The evicted Pod goes through the same terminating phase as in Drill
4. On size: by default an EndpointSlice holds at most 100 endpoints
(kube-controller-manager --max-endpoints-per-slice); larger
Services spread across several slices – so always query by the label
kubernetes.io/service-name, never by the slice name.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the names of all Pods that are not
readyin the slices ofweb. Target time: 60 s - Show how many disruptions PDB
webcurrently allows. Target time: 20 s - Create a headless Service
db(port 5432, selectorapp=db) that also publishes unready addresses. Target time: 60 s - Show where the default for endpoints per slice would be overridden on the kube-controller-manager. Target time: 30 s
Solution
k exec $P1 -- touch /tmp/down; sleep 8
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{.items[*].endpoints[?(@.conditions.ready==false)].targetRef.name}'; echo
k get pdb web -o jsonpath='{.status.disruptionsAllowed}'; echo
k create service clusterip db --clusterip=None --tcp=5432 $do > db.yaml
sed -i '/^ clusterIP: None/a\ publishNotReadyAddresses: true' db.yaml
k apply -f db.yaml
k get svc db -o jsonpath='{.spec.clusterIP} {.spec.publishNotReadyAddresses}'; echo
k -n kube-system get pod -l component=kube-controller-manager -o yaml \
| grep -c max-endpoints-per-slice
k exec $P1 -- rm /tmp/downExpected (with Pod $P1 made unready via
touch /tmp/down):
web-565f96c64-dtw9w
0
service/db created
None true
0
0 matches in the last command means: the flag is not
set, and the default of 100 applies.
Save your progress? Sign up for free
Checkpoint
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"Expected: two Pods with
r=true s=true t=false.
k get svc web-peers -o jsonpath='{.spec.publishNotReadyAddresses}'Expected: true
k get svc web -o jsonpath='{.spec.trafficDistribution}'Expected: PreferClose
k get pdb webExpected: ALLOWED DISRUPTIONS
1.
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$(k get svc web -o jsonpath='{.spec.clusterIP}'):"Expected: two backends, both
(active).
Challenge
Start a rollout of web while the client sends two
requests per second, and prove that not a single one fails.
Hint: k rollout restart returns
immediately – right afterwards, collect the response codes with
curl -w '%{http_code}' and count them.
Solution
k rollout restart deployment web
k exec client -- sh -c 'for i in $(seq 60); do
curl -s -o /dev/null -w "%{http_code}\n" -m1 http://web/hostname; sleep 0.5; done' \
| sort | uniq -c
k rollout status deployment web
k get pods -l app=webExpected:
deployment.apps/web restarted
60 200
deployment "web" successfully rolled out
NAME READY STATUS RESTARTS AGE
web-565f96c64-dtw9w 1/1 Terminating 0 3m11s
web-565f96c64-p2zj9 1/1 Terminating 0 64s
web-5b4dddcd9b-n7vbg 1/1 Running 0 30s
web-5b4dddcd9b-nzpx8 1/1 Running 0 34s
New Pods only get traffic once they are ready; old ones lose it immediately when terminating, but keep running for up to 60 s. That way, even a connection that a load balancer sent to the old Pod shortly before survives.
Save your progress? Sign up for free
Cleanup
The zone labels are cluster-wide and have to go:
k label node worker1 topology.kubernetes.io/zone-
k label node worker2 topology.kubernetes.io/zone-
k get nodes -L topology.kubernetes.io/zone
k delete namespace tag09
k config set-context --current --namespace=defaultExpected: column ZONE empty for all
three nodes, namespace "tag09" deleted.
Exam Tip
- “Service has no endpoints”: first
k get endpointslices -l kubernetes.io/service-name=…with conditions, then check the selector against the Pod labels andtargetPortagainst the container port. - The
ENDPOINTScolumn lies for unready Pods – read the conditions or, in Cilium,cilium-dbg service list(maintenance,terminating). - Docs to know: kubernetes.io “EndpointSlices”, “Traffic Distribution” (section in the Service article) and “Specifying a Disruption Budget for your Application”.
Enterprise Note
- Readiness probes check only what the Pod itself can do – no dependencies like the database, otherwise all endpoints drop out at once when it fails.
- A preStop pause plus grace period are mandatory behind external load balancers that only deregister targets after seconds (cloud LBs, ingress controllers outside).
trafficDistribution: PreferSameZonesaves cross-zone traffic costs in clouds – with Cilium only afterloadBalancer.serviceTopology=trueand with enough replicas per zone.- PDBs belong to every replicated service;
minAvailableequal to the replica count blocks every node update. - Very large Services (thousands of endpoints) create many slices and a lot of watch traffic – one reason to monitor endpoint counts and API server load.