Skip to content
Hands-on IT IT Drills

Day 9: Endpoints and Availability

Goal: You read EndpointSlices including their conditions, control with readiness, publishNotReadyAddresses, preStop and trafficDistribution which Pods get traffic, and verify it in Cilium. · Duration: approx. 70 min · Prerequisite: Day 6

Briefing

  • CKNE: Service Networking and DNS (25 %) – endpoints decide whether a Service answers at all; the most common reason for “the Service does not work”.
  • Every endpoint has three conditions: ready, serving, terminating. Cilium translates them into backend states (active, maintenance, terminating).
  • The backend’s readiness probe (agnhost netexec) checks test ! -e /tmp/down – that is how you toggle readiness with touch.
 Pod Ready? ── kubelet ──> Pod status ──> EndpointSlice controller ──> EndpointSlice
                                                                         │
                                         Cilium agent (eBPF backends) <──┘

Drill 1: Read EndpointSlices

mkdir -p ~/ckne/tag09 && cd ~/ckne/tag09
k create ns tag09
k config set-context --current --namespace=tag09
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: web
        image: registry.k8s.io/e2e-test-images/agnhost:2.66.1
        args: ["netexec", "--http-port=8080", "--delay-shutdown=45"]
        ports:
        - containerPort: 8080
        readinessProbe:
          exec:
            command: ["sh", "-c", "test ! -e /tmp/down"]
          periodSeconds: 2
        lifecycle:
          preStop:
            sleep:
              seconds: 15
EOF
k apply -f web.yaml
k expose deployment web --port=80 --target-port=8080
k run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k rollout status deployment web
k get pods -o wide
k get endpointslices -l kubernetes.io/service-name=web -o yaml

Expected (abridged):

deployment "web" successfully rolled out
NAME                  READY   STATUS    ...   IP           NODE
client                1/1     Running   ...   10.0.2.186   worker2
web-565f96c64-l72c9   1/1     Running   ...   10.0.1.47    worker1
web-565f96c64-n2pjp   1/1     Running   ...   10.0.2.214   worker2
- addressType: IPv4
  endpoints:
  - addresses:
    - 10.0.1.47
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: worker1
    targetRef:
      kind: Pod
      name: web-565f96c64-l72c9
  ...
  metadata:
    labels:
      endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
      kubernetes.io/service-name: web
    name: web-9ts49
    ownerReferences:
    - kind: Service
      name: web
  ports:
  - name: ""
    port: 8080
    protocol: TCP

The slice belongs to the Service (ownerReferences) and finds it via the label kubernetes.io/service-name. The port is the container port, not the Service port. To fit the three conditions on one line, you create a jsonpath template:

JP='{range .items[*].endpoints[*]}{.targetRef.name}{" r="}{.conditions.ready}'\
'{" s="}{.conditions.serving}{" t="}{.conditions.terminating}{"\n"}{end}'
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"

Expected:

web-565f96c64-l72c9 r=true s=true t=false
web-565f96c64-n2pjp r=true s=true t=false

Save your progress? Sign up for free

Drill 2: Readiness controls the endpoints

P1=$(k get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
k exec $P1 -- touch /tmp/down
sleep 8
k get pods -l app=web
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"
k get endpointslices -l kubernetes.io/service-name=web
for i in 1 2 3 4; do k exec client -- curl -s http://web/hostname; echo; done
CIP=$(k get svc web -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$CIP:"

Expected:

NAME                  READY   STATUS    RESTARTS   AGE
web-565f96c64-l72c9   0/1     Running   0          28s
web-565f96c64-n2pjp   1/1     Running   0          28s
web-565f96c64-l72c9 r=false s=false t=false
web-565f96c64-n2pjp r=true s=true t=false
NAME        ADDRESSTYPE   PORTS   ENDPOINTS              AGE
web-9ts49   IPv4          8080    10.0.1.47,10.0.2.214   29s
web-565f96c64-n2pjp
web-565f96c64-n2pjp
web-565f96c64-n2pjp
web-565f96c64-n2pjp
69   10.103.176.39:80/TCP      ClusterIP      1 => 10.0.1.47:8080/TCP (maintenance)
                                              2 => 10.0.2.214:8080/TCP (active)

After three failed probes (failureThreshold 3 × periodSeconds 2), the Pod is 0/1, the slice sets ready and serving to false, and Cilium parks the backend as maintenance. Trap: the ENDPOINTS column of k get endpointslices shows all addresses, including unready ones – only the conditions tell the truth. The Pod stays like this until Drill 3 removes the file again.

Save your progress? Sign up for free

Drill 3: publishNotReadyAddresses

Some services (database clusters, etcd) must find their peers before readiness. Two headless Services for comparison:

k expose deployment web --name=web-hl --cluster-ip=None --port=8080
k expose deployment web --name=web-peers --cluster-ip=None --port=8080 \
  $do > web-peers.yaml
sed -i '/^  clusterIP: None/a\  publishNotReadyAddresses: true' web-peers.yaml
grep -A1 clusterIP web-peers.yaml
k apply -f web-peers.yaml
k exec client -- dig +short web-hl.tag09.svc.cluster.local
k exec client -- dig +short web-peers.tag09.svc.cluster.local
k get endpointslices -l kubernetes.io/service-name=web-peers -o jsonpath="$JP"

Expected:

service/web-hl exposed
  clusterIP: None
  publishNotReadyAddresses: true
service/web-peers created
10.0.2.214
10.0.1.47
10.0.2.214
web-565f96c64-n2pjp r=true s=true t=false
web-565f96c64-l72c9 r=true s=false t=false

web-hl returns only the ready Pod, web-peers both. In the slice this shows as ready=true despite serving=false – the controller ignores readiness, as k explain service.spec.publishNotReadyAddresses describes. Now make the Pod healthy again:

k exec $P1 -- rm /tmp/down
sleep 4
k exec client -- dig +short web-hl.tag09.svc.cluster.local
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$CIP:"

Expected:

10.0.2.214
10.0.1.47
69   10.103.176.39:80/TCP      ClusterIP      1 => 10.0.1.47:8080/TCP (active)
                                              2 => 10.0.2.214:8080/TCP (active)

Save your progress? Sign up for free

Drill 4: Terminating endpoints, preStop and grace period

What happens to a Pod that is deleted while traffic is still coming in?

P1=$(k get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
PIP=$(k get pod $P1 -o jsonpath='{.status.podIP}')
k delete pod $P1 --wait=false
sleep 3
k get pods -l app=web
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A2 "$CIP:"
for i in 1 2 3 4; do k exec client -- curl -s http://web/hostname; echo; done
k exec client -- curl -s -m2 http://$PIP:8080/hostname; echo " <- direkt"

Expected:

pod "web-565f96c64-l72c9" deleted from tag09 namespace
NAME                  READY   STATUS        RESTARTS   AGE
web-565f96c64-dtw9w   1/1     Running       0          4s
web-565f96c64-l72c9   1/1     Terminating   0          69s
web-565f96c64-n2pjp   1/1     Running       0          69s
web-565f96c64-l72c9 r=false s=true t=true
web-565f96c64-n2pjp r=true s=true t=false
web-565f96c64-dtw9w r=true s=true t=false
69   10.103.176.39:80/TCP      ClusterIP      1 => 10.0.1.47:8080/TCP (terminating)
                                              2 => 10.0.1.120:8080/TCP (active)
                                              3 => 10.0.2.214:8080/TCP (active)
web-565f96c64-dtw9w
web-565f96c64-n2pjp
web-565f96c64-dtw9w
web-565f96c64-n2pjp
web-565f96c64-l72c9 <- direkt

Immediately on deletion, the endpoint becomes terminating and ready=false – new connections only go to the other Pods. serving=true means: the Pod could still answer, and it does, as the direct call shows. The timing: first 15 s of preStop.sleep, then SIGTERM, which agnhost bridges for another 45 s with --delay-shutdown=45 – together exactly the terminationGracePeriodSeconds: 60. In that time, Cilium agents and typical external load balancers have usually removed the endpoint. After a good minute:

k get pods -l app=web
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"

Expected:

NAME                  READY   STATUS    RESTARTS   AGE
web-565f96c64-dtw9w   1/1     Running   0          73s
web-565f96c64-n2pjp   1/1     Running   0          2m18s
web-565f96c64-n2pjp r=true s=true t=false
web-565f96c64-dtw9w r=true s=true t=false

Save your progress? Sign up for free

Drill 5: trafficDistribution and zones

k explain service.spec.trafficDistribution | sed -n '/DESCRIPTION/,$p'
k patch svc web -p '{"spec":{"trafficDistribution":"PreferBogus"}}'

Expected (abridged):

DESCRIPTION:
    TrafficDistribution offers a way to express preferences for how traffic is
    distributed to Service endpoints. ... If set
    to "PreferClose", implementations should prioritize endpoints that are in
    the same zone.
The Service "web" is invalid: spec.trafficDistribution: Unsupported value:
"PreferBogus": supported values: "PreferClose", "PreferSameZone", "PreferSameNode"

k explain only names PreferClose; which values 1.37 really knows is revealed by the validation. Zones come from the node label topology.kubernetes.io/zone – in the lab you set them yourself:

k label node worker1 topology.kubernetes.io/zone=zone-a
k label node worker2 topology.kubernetes.io/zone=zone-b
k get nodes -L topology.kubernetes.io/zone
k patch svc web -p '{"spec":{"trafficDistribution":"PreferSameZone"}}'
sleep 3
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{range .items[*].endpoints[*]}{.nodeName} {.zone} {.hints}{"\n"}{end}'
k patch svc web -p '{"spec":{"trafficDistribution":"PreferSameNode"}}'
sleep 3
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{range .items[*].endpoints[*]}{.nodeName} {.zone} {.hints}{"\n"}{end}'

Expected (abridged):

node/worker1 labeled
node/worker2 labeled
NAME      STATUS   ROLES           AGE   VERSION   ZONE
cp1       Ready    control-plane   53m   v1.37.2
worker1   Ready    <none>          47m   v1.37.2   zone-a
worker2   Ready    <none>          47m   v1.37.2   zone-b
service/web patched
worker2 zone-b {"forZones":[{"name":"zone-b"}]}
worker1 zone-a {"forZones":[{"name":"zone-a"}]}
service/web patched
worker2 zone-b {"forNodes":[{"name":"worker2"}],"forZones":[{"name":"zone-b"}]}
worker1 zone-a {"forNodes":[{"name":"worker1"}],"forZones":[{"name":"zone-a"}]}

The controller writes hints (hints) into the slice – nothing more. Whether they take effect is decided by the Service implementation. The client runs on worker2 (zone b):

for i in 1 2 3 4 5 6; do k exec client -- curl -s http://web/hostname; echo; done
cilium config view | grep service-topology

Expected:

web-565f96c64-n2pjp
web-565f96c64-n2pjp
web-565f96c64-dtw9w
web-565f96c64-dtw9w
web-565f96c64-n2pjp
web-565f96c64-n2pjp
enable-service-topology                           false

Both zones answer: Cilium only evaluates hints with the Helm value loadBalancer.serviceTopology=true (agent option enable-service-topology). In the lab that stays off – in the exam the rule is: first check whether the implementation honors hints before you hunt for “wrong” routing. PreferClose is the old name and now returns a warning:

k patch svc web -p '{"spec":{"trafficDistribution":"PreferClose"}}'

Expected:

Warning: spec.trafficDistribution: "PreferClose" is deprecated; use "PreferSameZone"
service/web patched

Save your progress? Sign up for free

Drill 6: PodDisruptionBudget – readiness counts

A PDB protects against voluntary disruptions (drain, eviction) and only counts ready Pods – the same readiness that controls the endpoints.

k create pdb web --selector=app=web --min-available=1
k get pdb web
P1=$(k get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
P2=$(k get pod -l app=web -o jsonpath='{.items[1].metadata.name}')
k exec $P1 -- touch /tmp/down
sleep 8
k get pdb web
cat > evict.json <<EOF
{"apiVersion":"policy/v1","kind":"Eviction","metadata":{"name":"$P2"}}
EOF
k create --raw /api/v1/namespaces/tag09/pods/$P2/eviction -f evict.json

Expected:

poddisruptionbudget.policy/web created
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    1               N/A               1                     2s
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    1               N/A               0                     11s
Error from server (TooManyRequests): Cannot evict pod as it would violate the pod's
disruption budget.

k drain sends exactly this eviction for every Pod – without practicing a node drain, you see here why a drain hangs: an unready Pod uses up the budget. Make the Pod healthy again, then it works:

k exec $P1 -- rm /tmp/down
sleep 5
k get pdb web
k create --raw /api/v1/namespaces/tag09/pods/$P2/eviction -f evict.json; echo
k get pods -l app=web

Expected:

NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    1               N/A               1                     17s
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","code":201}
NAME                  READY   STATUS        RESTARTS   AGE
web-565f96c64-dtw9w   1/1     Running       0          2m10s
web-565f96c64-n2pjp   1/1     Terminating   0          3m15s
web-565f96c64-p2zj9   1/1     Running       0          3s

The evicted Pod goes through the same terminating phase as in Drill 4. On size: by default an EndpointSlice holds at most 100 endpoints (kube-controller-manager --max-endpoints-per-slice); larger Services spread across several slices – so always query by the label kubernetes.io/service-name, never by the slice name.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the names of all Pods that are not ready in the slices of web. Target time: 60 s
  2. Show how many disruptions PDB web currently allows. Target time: 20 s
  3. Create a headless Service db (port 5432, selector app=db) that also publishes unready addresses. Target time: 60 s
  4. Show where the default for endpoints per slice would be overridden on the kube-controller-manager. Target time: 30 s

Solution

k exec $P1 -- touch /tmp/down; sleep 8
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{.items[*].endpoints[?(@.conditions.ready==false)].targetRef.name}'; echo
k get pdb web -o jsonpath='{.status.disruptionsAllowed}'; echo
k create service clusterip db --clusterip=None --tcp=5432 $do > db.yaml
sed -i '/^  clusterIP: None/a\  publishNotReadyAddresses: true' db.yaml
k apply -f db.yaml
k get svc db -o jsonpath='{.spec.clusterIP} {.spec.publishNotReadyAddresses}'; echo
k -n kube-system get pod -l component=kube-controller-manager -o yaml \
  | grep -c max-endpoints-per-slice
k exec $P1 -- rm /tmp/down

Expected (with Pod $P1 made unready via touch /tmp/down):

web-565f96c64-dtw9w
0
service/db created
None true
0

0 matches in the last command means: the flag is not set, and the default of 100 applies.

Save your progress? Sign up for free

Checkpoint

k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"

Expected: two Pods with r=true s=true t=false.

k get svc web-peers -o jsonpath='{.spec.publishNotReadyAddresses}'

Expected: true

k get svc web -o jsonpath='{.spec.trafficDistribution}'

Expected: PreferClose

k get pdb web

Expected: ALLOWED DISRUPTIONS 1.

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$(k get svc web -o jsonpath='{.spec.clusterIP}'):"

Expected: two backends, both (active).

Challenge

Start a rollout of web while the client sends two requests per second, and prove that not a single one fails.

Hint: k rollout restart returns immediately – right afterwards, collect the response codes with curl -w '%{http_code}' and count them.

Solution

k rollout restart deployment web
k exec client -- sh -c 'for i in $(seq 60); do
  curl -s -o /dev/null -w "%{http_code}\n" -m1 http://web/hostname; sleep 0.5; done' \
  | sort | uniq -c
k rollout status deployment web
k get pods -l app=web

Expected:

deployment.apps/web restarted
     60 200
deployment "web" successfully rolled out
NAME                   READY   STATUS        RESTARTS   AGE
web-565f96c64-dtw9w    1/1     Terminating   0          3m11s
web-565f96c64-p2zj9    1/1     Terminating   0          64s
web-5b4dddcd9b-n7vbg   1/1     Running       0          30s
web-5b4dddcd9b-nzpx8   1/1     Running       0          34s

New Pods only get traffic once they are ready; old ones lose it immediately when terminating, but keep running for up to 60 s. That way, even a connection that a load balancer sent to the old Pod shortly before survives.

Save your progress? Sign up for free

Cleanup

The zone labels are cluster-wide and have to go:

k label node worker1 topology.kubernetes.io/zone-
k label node worker2 topology.kubernetes.io/zone-
k get nodes -L topology.kubernetes.io/zone
k delete namespace tag09
k config set-context --current --namespace=default

Expected: column ZONE empty for all three nodes, namespace "tag09" deleted.

Exam Tip

  • “Service has no endpoints”: first k get endpointslices -l kubernetes.io/service-name=… with conditions, then check the selector against the Pod labels and targetPort against the container port.
  • The ENDPOINTS column lies for unready Pods – read the conditions or, in Cilium, cilium-dbg service list (maintenance, terminating).
  • Docs to know: kubernetes.io “EndpointSlices”, “Traffic Distribution” (section in the Service article) and “Specifying a Disruption Budget for your Application”.

Enterprise Note

  • Readiness probes check only what the Pod itself can do – no dependencies like the database, otherwise all endpoints drop out at once when it fails.
  • A preStop pause plus grace period are mandatory behind external load balancers that only deregister targets after seconds (cloud LBs, ingress controllers outside).
  • trafficDistribution: PreferSameZone saves cross-zone traffic costs in clouds – with Cilium only after loadBalancer.serviceTopology=true and with enough replicas per zone.
  • PDBs belong to every replicated service; minAvailable equal to the replica count blocks every node update.
  • Very large Services (thousands of endpoints) create many slices and a lot of watch traffic – one reason to monitor endpoint counts and API server load.