Skip to content
Hands-on IT IT Drills

Day 6: Services on Layer 4

Goal: You set up every Service type and every traffic policy imperatively, measure their effect with curl and read them in EndpointSlices and Cilium’s eBPF entries. · Duration: approx. 75 min · Prerequisite: Day 4

Briefing

  • CKNE: Service Networking and DNS (25 %) – Service types, session affinity, externalTrafficPolicy and internalTrafficPolicy are at the core.
  • Without kube-proxy, Cilium implements every Service as an entry in eBPF maps (cilium-dbg service list); it translates connections from Pods right at the socket.
  • The backend is agnhost netexec: /hostname returns the Pod name, /clientip the source address that arrives at the Pod.
 router .123 ── NodePort :30080 ──> cp1 / worker1 / worker2 ──┐
 tag06/client (worker1) ── ClusterIP echo:80 ─────────────────┴─> echo-… :8080
                                                                  (worker1, worker2)

Drill 1: ClusterIP and EndpointSlice

mkdir -p ~/ckne/tag06 && cd ~/ckne/tag06
k create ns tag06
k config set-context --current --namespace=tag06
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
  --replicas=2 --port=8080 -- /agnhost netexec --http-port=8080
k run client --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k rollout status deployment echo
k wait --for=condition=Ready pod/client
k get pods -o wide
k expose deployment echo --port=80 --target-port=8080
k get endpointslices -l kubernetes.io/service-name=echo
for i in 1 2 3 4; do k exec client -- curl -s http://echo/hostname; echo; done

Expected (abridged):

deployment "echo" successfully rolled out
pod/client condition met
NAME                   READY   STATUS    ...   IP           NODE
client                 1/1     Running   ...   10.0.1.115   worker1
echo-cc8555f69-cvn8f   1/1     Running   ...   10.0.1.166   worker1
echo-cc8555f69-kc99p   1/1     Running   ...   10.0.2.114   worker2
service/echo exposed
NAME         ADDRESSTYPE   PORTS   ENDPOINTS               AGE
echo-6lbgk   IPv4          8080    10.0.2.114,10.0.1.166   0s
echo-cc8555f69-kc99p
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-kc99p

nodeName pins the client to worker1 – you need that in Drill 7. The Service itself knows no Pods: the EndpointSlice controller collects the Pod IPs for the selector app=echo, Cilium reads the slices and writes the eBPF entries:

CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$CIP:"

Expected:

28   10.97.228.80:80/TCP     ClusterIP      1 => 10.0.1.166:8080/TCP (active)
                                            2 => 10.0.2.114:8080/TCP (active)

Save your progress? Sign up for free

Drill 2: NodePort with a fixed port

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
  | grep 'NodePort:'
k expose deployment echo --name=echo-np --type=NodePort --port=80 --target-port=8080
k get svc echo-np
k patch svc echo-np -p '{"spec":{"ports":[{"port":80,"nodePort":8080}]}}'
k patch svc echo-np -p '{"spec":{"ports":[{"port":80,"nodePort":30080}]}}'
k get svc echo-np

Expected (abridged):

  - NodePort:       Enabled (Range: 30000-32767)
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
echo-np   NodePort   10.100.165.152   <none>        80:30152/TCP   0s
The Service "echo-np" is invalid: spec.ports[0].nodePort: Invalid value: 8080: provided
port is not in the valid range. The range of valid ports is 30000-32767
service/echo-np patched
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
echo-np   NodePort   10.100.165.152   <none>        80:30080/TCP   1s

The API server determines the range (--service-node-port-range, default 30000–32767). Cilium has its own range (nodePort.range, same default) – if you change it, adjust both. The patch hits the port entry via its key port: 80. Now from router, that is, from outside the cluster:

ssh [email protected]
for n in 172.16.0.126 172.16.0.228 172.16.0.83; do
  curl -s -m3 http://$n:30080/hostname; echo " <- $n"
done
exit
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 ':30080/'

Expected:

echo-cc8555f69-cvn8f <- 172.16.0.126
echo-cc8555f69-kc99p <- 172.16.0.228
echo-cc8555f69-kc99p <- 172.16.0.83
32   0.0.0.0:30080/TCP       NodePort       1 => 10.0.1.166:8080/TCP (active)
                                            2 => 10.0.2.114:8080/TCP (active)

Every node answers, even cp1 without an echo Pod. 0.0.0.0:30080 means: on all node addresses.

Save your progress? Sign up for free

Drill 3: LoadBalancer without an IP

k expose deployment echo --name=echo-lb --type=LoadBalancer --port=80 \
  --target-port=8080
k get svc echo-lb
k get svc echo-lb -o jsonpath='{.status}'; echo
cilium config view | grep -E 'enable-lb-ipam|default-lb-service-ipam'
k get ciliumloadbalancerippools

Expected:

service/echo-lb exposed
NAME      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
echo-lb   LoadBalancer   10.103.31.143   <pending>     80:32063/TCP   6s
{"loadBalancer":{}}
default-lb-service-ipam                           lbipam
enable-lb-ipam                                    true
No resources found

A LoadBalancer Service is a NodePort Service plus the request “give me an external IP”. In the cloud, the cloud controller manager fulfills it; here it would be Cilium’s LB IPAM – which is switched on but has no pool. So it stays <pending>, but NodePort 32063 works anyway. Day 15 creates the pool and announces the IPs via BGP and L2.

NP=$(k get svc echo-lb -o jsonpath='{.spec.ports[0].nodePort}')
curl -s -m3 http://172.16.0.228:$NP/hostname; echo

Expected: a Pod name such as echo-cc8555f69-kc99p.

Save your progress? Sign up for free

Drill 4: Headless and ExternalName – Services in DNS only

k expose deployment echo --name=echo-hl --cluster-ip=None --port=8080
k get svc echo-hl
k exec client -- dig +short echo-hl.tag06.svc.cluster.local
k exec client -- dig +short echo.tag06.svc.cluster.local
k exec client -- dig +short -t SRV echo-hl.tag06.svc.cluster.local
k exec client -- dig +short 10-0-1-166.echo-hl.tag06.svc.cluster.local
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -E ':8080/TCP +ClusterIP'

Expected:

service/echo-hl exposed
NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
echo-hl   ClusterIP   None         <none>        8080/TCP   0s
10.0.1.166
10.0.2.114
10.97.228.80
0 50 8080 10-0-2-114.echo-hl.tag06.svc.cluster.local.
0 50 8080 10-0-1-166.echo-hl.tag06.svc.cluster.local.
10.0.1.166

Headless returns the Pod IPs as A records, the normal Service only its ClusterIP. There is no virtual IP – and therefore no eBPF entry either (the last grep stays empty); the client distributes the load itself. A StatefulSet, for example, needs this for stable names per Pod. ExternalName is even thinner – a CNAME:

k create service externalname docs --external-name=kubernetes.io
k get svc docs
k exec client -- dig docs.tag06.svc.cluster.local +noall +answer
k exec client -- curl -sI -m5 http://docs | head -1
k exec client -- curl -sI -m5 -H 'Host: kubernetes.io' http://docs | head -1

Expected:

service/docs created
NAME   TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)   AGE
docs   ExternalName   <none>       kubernetes.io   <none>    1s
docs.tag06.svc.cluster.local. 30 IN CNAME   kubernetes.io.
kubernetes.io.      30  IN  A   3.33.186.135
kubernetes.io.      30  IN  A   15.197.167.90
HTTP/1.1 400 Bad Request
HTTP/1.1 301 Moved Permanently

Why 400? The CNAME only changes name resolution, not the HTTP Host header – the web server gets Host: docs and rejects it. With TLS, the certificate does not match for the same reason.

Save your progress? Sign up for free

Drill 5: sessionAffinity ClientIP

k patch svc echo -p '{"spec":{"sessionAffinity":"ClientIP"}}'
k get svc echo -o jsonpath='{.spec.sessionAffinityConfig}'; echo
for i in 1 2 3 4 5 6; do k exec client -- curl -s http://echo/hostname; echo; done
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf lb list \
  | grep "$CIP:80/TCP (0)"
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg map list | grep lb4_affinity

Expected:

service/echo patched
{"clientIP":{"timeoutSeconds":10800}}
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
10.97.228.80:80/TCP (0)   0.0.0.0:0 (30) (0) [ClusterIP, sessionAffinity, non-routable]
cilium_lb4_affinity        1             0            true

All requests land on the same Pod, for three hours (default timeout 10800 s). The mapping “client IP → backend” lives in the map cilium_lb4_affinity of the node where the client runs – hence the agent on worker1. Affinity is tied to the source IP: behind a NAT, many clients share one Pod.

Save your progress? Sign up for free

Drill 6: externalTrafficPolicy Cluster and Local

What does the Pod see as the source when router calls the NodePort?

k get ciliumnodes -o custom-columns='NAME:.metadata.name,'\
'HOSTIP:.spec.addresses[?(@.type=="CiliumInternalIP")].ip'
ssh [email protected]
for n in 172.16.0.126 172.16.0.228 172.16.0.83; do
  for i in 1 2 3; do curl -s -m3 http://$n:30080/clientip; echo " <- $n"; done
done
exit

Expected:

NAME      HOSTIP
cp1       10.0.0.88
worker1   10.0.1.116
worker2   10.0.2.241
10.0.0.88:52892 <- 172.16.0.126
10.0.0.88:52904 <- 172.16.0.126
10.0.0.88:52912 <- 172.16.0.126
172.16.0.123:37066 <- 172.16.0.228
172.16.0.123:37072 <- 172.16.0.228
172.16.0.123:37088 <- 172.16.0.228
10.0.2.241:58604 <- 172.16.0.83
172.16.0.123:58610 <- 172.16.0.83
10.0.2.241:58622 <- 172.16.0.83

With Cluster (default), every node chooses among all backends. If the backend is on another node, Cilium replaces the source with its own cilium_host address (SNAT) so the reply finds its way back – the real client IP is gone. Only when the backend happens to be local does 172.16.0.123 arrive.

k patch svc echo-np -p '{"spec":{"externalTrafficPolicy":"Local"}}'
ssh [email protected]
for n in 172.16.0.126 172.16.0.228 172.16.0.83; do
  for i in 1 2; do curl -sS -m3 http://$n:30080/clientip; echo " <- $n"; done
done
exit
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=cp1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
  | grep -A1 ':30080/'

Expected (abridged):

service/echo-np patched
curl: (7) Failed to connect to 172.16.0.126 port 30080 after 1 ms: Couldn't connect ...
 <- 172.16.0.126
curl: (7) Failed to connect to 172.16.0.126 port 30080 after 0 ms: Couldn't connect ...
 <- 172.16.0.126
172.16.0.123:37104 <- 172.16.0.228
172.16.0.123:37110 <- 172.16.0.228
172.16.0.123:58624 <- 172.16.0.83
172.16.0.123:58636 <- 172.16.0.83
32   0.0.0.0:30080/TCP       NodePort
--
37   0.0.0.0:30080/TCP/i     NodePort       1 => 10.0.1.166:8080/TCP (active)
                                            2 => 10.0.2.114:8080/TCP (active)

Local preserves the client IP, but only nodes with their own backend answer. For this, Cilium creates two entries: the external one without backends on cp1 (connection refused) and /i for cluster-internal calls, which still use all backends. An external load balancer therefore has to know which nodes have backends – more on that in the Challenge.

Save your progress? Sign up for free

Drill 7: internalTrafficPolicy Local

k run client2 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k wait --for=condition=Ready pod/client2
k expose deployment echo --name=echo-local --port=80 --target-port=8080 \
  $do > echo-local.yaml
sed -i '/^  ports:/i\  internalTrafficPolicy: Local' echo-local.yaml
grep -B1 -A1 internalTrafficPolicy echo-local.yaml
k apply -f echo-local.yaml
for p in client client2; do
  for i in 1 2 3; do
    k exec $p -- curl -s http://echo-local/hostname; echo " <- $p"
  done
done
LIP=$(k get svc echo-local -o jsonpath='{.spec.clusterIP}')
curl -sS -m3 http://$LIP/hostname

Expected:

spec:
  internalTrafficPolicy: Local
  ports:
service/echo-local created
echo-cc8555f69-cvn8f <- client
echo-cc8555f69-cvn8f <- client
echo-cc8555f69-cvn8f <- client
echo-cc8555f69-kc99p <- client2
echo-cc8555f69-kc99p <- client2
echo-cc8555f69-kc99p <- client2
curl: (7) Failed to connect to 10.98.87.203 port 80 after 0 ms: Couldn't connect to ...

Each client stays on its node; the last curl runs on cp1 – there is no backend there, so no connection. Good for node-local services (log agent, DNS cache), dangerous for everything else. A look into the agents on worker1 and cp1 shows it:

CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$LIP:"
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=cp1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$LIP:"

Expected:

42   10.98.87.203:80/TCP     ClusterIP      1 => 10.0.1.166:8080/TCP (active)
40   10.98.87.203:80/TCP     ClusterIP

Save your progress? Sign up for free

Drill 8: Service without a selector and your own EndpointSlice

This is how you bring a service outside the cluster in under a cluster name. On router, start a test server in a second terminal:

mkdir -p ~/tag06 && cd ~/tag06
echo "hallo vom router" > index.html
python3 -m http.server 8000

Back on cp1:

k create service clusterip router --tcp=80:8000 $do > router-svc.yaml
sed -i '/selector:/,+1d' router-svc.yaml
cat > router-eps.yaml <<'EOF'
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: router-1
  labels:
    kubernetes.io/service-name: router
    endpointslice.kubernetes.io/managed-by: tag06.example.com
addressType: IPv4
ports:
- name: 80-8000
  protocol: TCP
  port: 8000
endpoints:
- addresses:
  - 172.16.0.123
EOF
k apply -f router-svc.yaml -f router-eps.yaml
k get svc router -o wide
k get endpointslices -l kubernetes.io/service-name=router
k exec client -- curl -s -m3 http://router/
RIP=$(k get svc router -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$RIP:"

Expected:

service/router created
endpointslice.discovery.k8s.io/router-1 created
NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
router   ClusterIP   10.106.226.85   <none>        80/TCP    0s    <none>
NAME       ADDRESSTYPE   PORTS   ENDPOINTS      AGE
router-1   IPv4          8000    172.16.0.123   0s
hallo vom router
43   10.106.226.85:80/TCP    ClusterIP      1 => 172.16.0.123:8000/TCP (active)

Three things must match: the label kubernetes.io/service-name connects slice and Service, the port name in the slice must equal the one in the Service (80-8000, that is how k create service names it), and managed-by must not be endpointslice-controller.k8s.io, otherwise the controller removes your slice. Removing the selector after the fact leaves a stale slice from the controller behind – so create it without a selector right away.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Create a NodePort Service web for deployment/echo, port 80 → 8080, NodePort fixed at 31000. Target time: 60 s
  2. Limit the session affinity of Service echo to 60 seconds. Target time: 30 s
  3. Show Pod IP, Pod name and node of every endpoint of echo, one line each. Target time: 60 s
  4. Show type, externalTrafficPolicy and internalTrafficPolicy for all Services in tag06. Target time: 45 s

Solution

k expose deployment echo --name=web --type=NodePort --port=80 --target-port=8080
k patch svc web -p '{"spec":{"ports":[{"port":80,"nodePort":31000}]}}'
curl -s -m3 http://172.16.0.228:31000/hostname; echo
k patch svc echo -p \
  '{"spec":{"sessionAffinityConfig":{"clientIP":{"timeoutSeconds":60}}}}'
k get svc echo -o jsonpath='{.spec.sessionAffinityConfig}'; echo
k get endpointslices -l kubernetes.io/service-name=echo -o jsonpath=\
'{range .items[*].endpoints[*]}{.addresses[0]} {.targetRef.name} {.nodeName}{"\n"}{end}'
k get svc -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,\
ETP:.spec.externalTrafficPolicy,ITP:.spec.internalTrafficPolicy

Expected (abridged):

echo-cc8555f69-kc99p
{"clientIP":{"timeoutSeconds":60}}
10.0.2.114 echo-cc8555f69-kc99p worker2
10.0.1.166 echo-cc8555f69-cvn8f worker1
NAME         TYPE           ETP       ITP
docs         ExternalName   <none>    <none>
echo         ClusterIP      <none>    Cluster
echo-hl      ClusterIP      <none>    Cluster
echo-lb      LoadBalancer   Cluster   Cluster
echo-local   ClusterIP      <none>    Local
echo-np      NodePort       Local     Cluster
router       ClusterIP      <none>    Cluster
web          NodePort       Cluster   Cluster

If the first curl right after the patch returns nothing, wait a second – Cilium needs a moment for the new entry.

Save your progress? Sign up for free

Checkpoint

k get svc echo-np -o jsonpath='{.spec.ports[0].nodePort} {.spec.externalTrafficPolicy}'

Expected: 30080 Local

ssh [email protected] curl -s -m3 http://172.16.0.228:30080/clientip

Expected: 172.16.0.123:<port> – the real client IP.

k get svc echo-lb

Expected: EXTERNAL-IP <pending> (no LB IPAM pool until Day 15).

k exec client -- dig +short echo-hl.tag06.svc.cluster.local

Expected: the two Pod IPs of echo, no ClusterIP.

k exec client -- curl -s -m3 http://router/

Expected: hallo vom router

Challenge

Set echo-lb to externalTrafficPolicy: Local and show from router how an external load balancer recognizes which nodes it may send traffic to.

Hint: With Local, Kubernetes assigns an additional field spec.healthCheckNodePort to LoadBalancer Services.

Solution

k patch svc echo-lb -p '{"spec":{"externalTrafficPolicy":"Local"}}'
HC=$(k get svc echo-lb -o jsonpath='{.spec.healthCheckNodePort}'); echo $HC
for n in 172.16.0.126 172.16.0.228 172.16.0.83; do
  ssh [email protected] "curl -s -m3 -w ' %{http_code}' http://$n:$HC/"
  echo " <- $n"
done

Expected:

31601
{"service":{"namespace":"tag06","name":"echo-lb"},"localEndpoints":0}
 503 <- 172.16.0.126
{"service":{"namespace":"tag06","name":"echo-lb"},"localEndpoints":1}
 200 <- 172.16.0.228
{"service":{"namespace":"tag06","name":"echo-lb"},"localEndpoints":1}
 200 <- 172.16.0.83

Cilium answers the health check port itself (instead of kube-proxy): 200 with local backends, 503 without. Cloud load balancers check exactly this port and take cp1 out of rotation. A NodePort Service does not get the field – echo-np does not have it.

Save your progress? Sign up for free

Cleanup

On router, stop the test server with Ctrl+C, then:

rm -r ~/tag06
exit

On cp1:

k delete namespace tag06
k config set-context --current --namespace=default

Day 6 changed nothing cluster-wide.

Exam Tip

  • k expose knows --type, --port, --target-port, --cluster-ip=None, but no fixed NodePort and no policies – you set those with k patch or via a $do file.
  • externalTrafficPolicy only affects NodePort/LoadBalancer, internalTrafficPolicy only traffic from inside the cluster; Local without a local backend means: no answer.
  • Docs: kubernetes.io “Service”, “Service Internal Traffic Policy”, “Create an External Load Balancer” (section “Preserving the client source IP”).

Enterprise Note

  • externalTrafficPolicy: Local is common when backends need the real client IP (WAF, rate limits, audit) – then spread the Pods across the nodes (topology spread).
  • Instead of SNAT, Cilium can preserve the client IP even with Cluster using DSR (Direct Server Return); more on Day 7.
  • Session affinity by client IP is a stopgap; stateful apps are better served by cookies at the gateway (Gateway API sessionPersistence) or a shared session store.
  • Use ExternalName only for targets without TLS name verification or with a matching SAN – otherwise your own slice pointing to fixed IPs or an egress gateway (Day 16).
  • Manually maintained EndpointSlices go stale easily; in production a controller maintains them, or a ServiceEntry-like solution in the mesh.