Day 6: Services on Layer 4
Goal: You set up every Service type and every traffic policy imperatively, measure their effect with curl and read them in EndpointSlices and Cilium’s eBPF entries. · Duration: approx. 75 min · Prerequisite: Day 4
Briefing
- CKNE: Service Networking and DNS (25 %) – Service types, session
affinity,
externalTrafficPolicyandinternalTrafficPolicyare at the core. - Without kube-proxy, Cilium implements every Service as an entry in
eBPF maps (
cilium-dbg service list); it translates connections from Pods right at the socket. - The backend is agnhost
netexec:/hostnamereturns the Pod name,/clientipthe source address that arrives at the Pod.
router .123 ── NodePort :30080 ──> cp1 / worker1 / worker2 ──┐
tag06/client (worker1) ── ClusterIP echo:80 ─────────────────┴─> echo-… :8080
(worker1, worker2)
Drill 1: ClusterIP and EndpointSlice
mkdir -p ~/ckne/tag06 && cd ~/ckne/tag06
k create ns tag06
k config set-context --current --namespace=tag06
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
--replicas=2 --port=8080 -- /agnhost netexec --http-port=8080
k run client --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k rollout status deployment echo
k wait --for=condition=Ready pod/client
k get pods -o wide
k expose deployment echo --port=80 --target-port=8080
k get endpointslices -l kubernetes.io/service-name=echo
for i in 1 2 3 4; do k exec client -- curl -s http://echo/hostname; echo; doneExpected (abridged):
deployment "echo" successfully rolled out
pod/client condition met
NAME READY STATUS ... IP NODE
client 1/1 Running ... 10.0.1.115 worker1
echo-cc8555f69-cvn8f 1/1 Running ... 10.0.1.166 worker1
echo-cc8555f69-kc99p 1/1 Running ... 10.0.2.114 worker2
service/echo exposed
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
echo-6lbgk IPv4 8080 10.0.2.114,10.0.1.166 0s
echo-cc8555f69-kc99p
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-kc99p
nodeName pins the client to worker1 – you
need that in Drill 7. The Service itself knows no Pods: the
EndpointSlice controller collects the Pod IPs for the selector
app=echo, Cilium reads the slices and writes the eBPF
entries:
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$CIP:"Expected:
28 10.97.228.80:80/TCP ClusterIP 1 => 10.0.1.166:8080/TCP (active)
2 => 10.0.2.114:8080/TCP (active)
Save your progress? Sign up for free
Drill 2: NodePort with a fixed port
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
| grep 'NodePort:'
k expose deployment echo --name=echo-np --type=NodePort --port=80 --target-port=8080
k get svc echo-np
k patch svc echo-np -p '{"spec":{"ports":[{"port":80,"nodePort":8080}]}}'
k patch svc echo-np -p '{"spec":{"ports":[{"port":80,"nodePort":30080}]}}'
k get svc echo-npExpected (abridged):
- NodePort: Enabled (Range: 30000-32767)
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-np NodePort 10.100.165.152 <none> 80:30152/TCP 0s
The Service "echo-np" is invalid: spec.ports[0].nodePort: Invalid value: 8080: provided
port is not in the valid range. The range of valid ports is 30000-32767
service/echo-np patched
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-np NodePort 10.100.165.152 <none> 80:30080/TCP 1s
The API server determines the range
(--service-node-port-range, default 30000–32767). Cilium
has its own range (nodePort.range, same default) – if you
change it, adjust both. The patch hits the port entry via its key
port: 80. Now from router, that is, from
outside the cluster:
ssh [email protected]
for n in 172.16.0.126 172.16.0.228 172.16.0.83; do
curl -s -m3 http://$n:30080/hostname; echo " <- $n"
done
exit
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 ':30080/'Expected:
echo-cc8555f69-cvn8f <- 172.16.0.126
echo-cc8555f69-kc99p <- 172.16.0.228
echo-cc8555f69-kc99p <- 172.16.0.83
32 0.0.0.0:30080/TCP NodePort 1 => 10.0.1.166:8080/TCP (active)
2 => 10.0.2.114:8080/TCP (active)
Every node answers, even cp1 without an echo Pod.
0.0.0.0:30080 means: on all node addresses.
Save your progress? Sign up for free
Drill 3: LoadBalancer without an IP
k expose deployment echo --name=echo-lb --type=LoadBalancer --port=80 \
--target-port=8080
k get svc echo-lb
k get svc echo-lb -o jsonpath='{.status}'; echo
cilium config view | grep -E 'enable-lb-ipam|default-lb-service-ipam'
k get ciliumloadbalancerippoolsExpected:
service/echo-lb exposed
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-lb LoadBalancer 10.103.31.143 <pending> 80:32063/TCP 6s
{"loadBalancer":{}}
default-lb-service-ipam lbipam
enable-lb-ipam true
No resources found
A LoadBalancer Service is a NodePort Service plus the request “give
me an external IP”. In the cloud, the cloud controller manager fulfills
it; here it would be Cilium’s LB IPAM – which is switched on but has no
pool. So it stays <pending>, but NodePort 32063 works
anyway. Day 15 creates the pool and announces the IPs via BGP and
L2.
NP=$(k get svc echo-lb -o jsonpath='{.spec.ports[0].nodePort}')
curl -s -m3 http://172.16.0.228:$NP/hostname; echoExpected: a Pod name such as
echo-cc8555f69-kc99p.
Save your progress? Sign up for free
Drill 4: Headless and ExternalName – Services in DNS only
k expose deployment echo --name=echo-hl --cluster-ip=None --port=8080
k get svc echo-hl
k exec client -- dig +short echo-hl.tag06.svc.cluster.local
k exec client -- dig +short echo.tag06.svc.cluster.local
k exec client -- dig +short -t SRV echo-hl.tag06.svc.cluster.local
k exec client -- dig +short 10-0-1-166.echo-hl.tag06.svc.cluster.local
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -E ':8080/TCP +ClusterIP'Expected:
service/echo-hl exposed
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-hl ClusterIP None <none> 8080/TCP 0s
10.0.1.166
10.0.2.114
10.97.228.80
0 50 8080 10-0-2-114.echo-hl.tag06.svc.cluster.local.
0 50 8080 10-0-1-166.echo-hl.tag06.svc.cluster.local.
10.0.1.166
Headless returns the Pod IPs as A records, the normal Service only
its ClusterIP. There is no virtual IP – and therefore no eBPF entry
either (the last grep stays empty); the client distributes
the load itself. A StatefulSet, for example, needs this for stable names
per Pod. ExternalName is even thinner – a CNAME:
k create service externalname docs --external-name=kubernetes.io
k get svc docs
k exec client -- dig docs.tag06.svc.cluster.local +noall +answer
k exec client -- curl -sI -m5 http://docs | head -1
k exec client -- curl -sI -m5 -H 'Host: kubernetes.io' http://docs | head -1Expected:
service/docs created
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
docs ExternalName <none> kubernetes.io <none> 1s
docs.tag06.svc.cluster.local. 30 IN CNAME kubernetes.io.
kubernetes.io. 30 IN A 3.33.186.135
kubernetes.io. 30 IN A 15.197.167.90
HTTP/1.1 400 Bad Request
HTTP/1.1 301 Moved Permanently
Why 400? The CNAME only changes name resolution, not the HTTP Host
header – the web server gets Host: docs and rejects it.
With TLS, the certificate does not match for the same reason.
Save your progress? Sign up for free
Drill 5: sessionAffinity ClientIP
k patch svc echo -p '{"spec":{"sessionAffinity":"ClientIP"}}'
k get svc echo -o jsonpath='{.spec.sessionAffinityConfig}'; echo
for i in 1 2 3 4 5 6; do k exec client -- curl -s http://echo/hostname; echo; done
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf lb list \
| grep "$CIP:80/TCP (0)"
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg map list | grep lb4_affinityExpected:
service/echo patched
{"clientIP":{"timeoutSeconds":10800}}
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
echo-cc8555f69-cvn8f
10.97.228.80:80/TCP (0) 0.0.0.0:0 (30) (0) [ClusterIP, sessionAffinity, non-routable]
cilium_lb4_affinity 1 0 true
All requests land on the same Pod, for three hours (default timeout
10800 s). The mapping “client IP → backend” lives in the map
cilium_lb4_affinity of the node where the client runs –
hence the agent on worker1. Affinity is tied to the source
IP: behind a NAT, many clients share one Pod.
Save your progress? Sign up for free
Drill 6: externalTrafficPolicy Cluster and Local
What does the Pod see as the source when router calls
the NodePort?
k get ciliumnodes -o custom-columns='NAME:.metadata.name,'\
'HOSTIP:.spec.addresses[?(@.type=="CiliumInternalIP")].ip'
ssh [email protected]
for n in 172.16.0.126 172.16.0.228 172.16.0.83; do
for i in 1 2 3; do curl -s -m3 http://$n:30080/clientip; echo " <- $n"; done
done
exitExpected:
NAME HOSTIP
cp1 10.0.0.88
worker1 10.0.1.116
worker2 10.0.2.241
10.0.0.88:52892 <- 172.16.0.126
10.0.0.88:52904 <- 172.16.0.126
10.0.0.88:52912 <- 172.16.0.126
172.16.0.123:37066 <- 172.16.0.228
172.16.0.123:37072 <- 172.16.0.228
172.16.0.123:37088 <- 172.16.0.228
10.0.2.241:58604 <- 172.16.0.83
172.16.0.123:58610 <- 172.16.0.83
10.0.2.241:58622 <- 172.16.0.83
With Cluster (default), every node chooses among all
backends. If the backend is on another node, Cilium replaces the source
with its own cilium_host address (SNAT) so the reply finds
its way back – the real client IP is gone. Only when the backend happens
to be local does 172.16.0.123 arrive.
k patch svc echo-np -p '{"spec":{"externalTrafficPolicy":"Local"}}'
ssh [email protected]
for n in 172.16.0.126 172.16.0.228 172.16.0.83; do
for i in 1 2; do curl -sS -m3 http://$n:30080/clientip; echo " <- $n"; done
done
exit
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=cp1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
| grep -A1 ':30080/'Expected (abridged):
service/echo-np patched
curl: (7) Failed to connect to 172.16.0.126 port 30080 after 1 ms: Couldn't connect ...
<- 172.16.0.126
curl: (7) Failed to connect to 172.16.0.126 port 30080 after 0 ms: Couldn't connect ...
<- 172.16.0.126
172.16.0.123:37104 <- 172.16.0.228
172.16.0.123:37110 <- 172.16.0.228
172.16.0.123:58624 <- 172.16.0.83
172.16.0.123:58636 <- 172.16.0.83
32 0.0.0.0:30080/TCP NodePort
--
37 0.0.0.0:30080/TCP/i NodePort 1 => 10.0.1.166:8080/TCP (active)
2 => 10.0.2.114:8080/TCP (active)
Local preserves the client IP, but only nodes with their
own backend answer. For this, Cilium creates two entries: the external
one without backends on cp1 (connection refused) and
/i for cluster-internal calls, which still use all
backends. An external load balancer therefore has to know which nodes
have backends – more on that in the Challenge.
Save your progress? Sign up for free
Drill 7: internalTrafficPolicy Local
k run client2 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k wait --for=condition=Ready pod/client2
k expose deployment echo --name=echo-local --port=80 --target-port=8080 \
$do > echo-local.yaml
sed -i '/^ ports:/i\ internalTrafficPolicy: Local' echo-local.yaml
grep -B1 -A1 internalTrafficPolicy echo-local.yaml
k apply -f echo-local.yaml
for p in client client2; do
for i in 1 2 3; do
k exec $p -- curl -s http://echo-local/hostname; echo " <- $p"
done
done
LIP=$(k get svc echo-local -o jsonpath='{.spec.clusterIP}')
curl -sS -m3 http://$LIP/hostnameExpected:
spec:
internalTrafficPolicy: Local
ports:
service/echo-local created
echo-cc8555f69-cvn8f <- client
echo-cc8555f69-cvn8f <- client
echo-cc8555f69-cvn8f <- client
echo-cc8555f69-kc99p <- client2
echo-cc8555f69-kc99p <- client2
echo-cc8555f69-kc99p <- client2
curl: (7) Failed to connect to 10.98.87.203 port 80 after 0 ms: Couldn't connect to ...
Each client stays on its node; the last curl runs on
cp1 – there is no backend there, so no connection. Good for
node-local services (log agent, DNS cache), dangerous for everything
else. A look into the agents on worker1 and
cp1 shows it:
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$LIP:"
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=cp1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$LIP:"Expected:
42 10.98.87.203:80/TCP ClusterIP 1 => 10.0.1.166:8080/TCP (active)
40 10.98.87.203:80/TCP ClusterIP
Save your progress? Sign up for free
Drill 8: Service without a selector and your own EndpointSlice
This is how you bring a service outside the cluster in under a
cluster name. On router, start a test server in a
second terminal:
mkdir -p ~/tag06 && cd ~/tag06
echo "hallo vom router" > index.html
python3 -m http.server 8000Back on cp1:
k create service clusterip router --tcp=80:8000 $do > router-svc.yaml
sed -i '/selector:/,+1d' router-svc.yaml
cat > router-eps.yaml <<'EOF'
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: router-1
labels:
kubernetes.io/service-name: router
endpointslice.kubernetes.io/managed-by: tag06.example.com
addressType: IPv4
ports:
- name: 80-8000
protocol: TCP
port: 8000
endpoints:
- addresses:
- 172.16.0.123
EOF
k apply -f router-svc.yaml -f router-eps.yaml
k get svc router -o wide
k get endpointslices -l kubernetes.io/service-name=router
k exec client -- curl -s -m3 http://router/
RIP=$(k get svc router -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$RIP:"Expected:
service/router created
endpointslice.discovery.k8s.io/router-1 created
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
router ClusterIP 10.106.226.85 <none> 80/TCP 0s <none>
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
router-1 IPv4 8000 172.16.0.123 0s
hallo vom router
43 10.106.226.85:80/TCP ClusterIP 1 => 172.16.0.123:8000/TCP (active)
Three things must match: the label
kubernetes.io/service-name connects slice and Service, the
port name in the slice must equal the one in the
Service (80-8000, that is how k create service
names it), and managed-by must not be
endpointslice-controller.k8s.io, otherwise the controller
removes your slice. Removing the selector after the fact leaves a stale
slice from the controller behind – so create it without a selector right
away.
Save your progress? Sign up for free
Exercise: Speed Drills
- Create a NodePort Service
webfordeployment/echo, port 80 → 8080, NodePort fixed at 31000. Target time: 60 s - Limit the session affinity of Service
echoto 60 seconds. Target time: 30 s - Show Pod IP, Pod name and node of every endpoint of
echo, one line each. Target time: 60 s - Show type,
externalTrafficPolicyandinternalTrafficPolicyfor all Services intag06. Target time: 45 s
Solution
k expose deployment echo --name=web --type=NodePort --port=80 --target-port=8080
k patch svc web -p '{"spec":{"ports":[{"port":80,"nodePort":31000}]}}'
curl -s -m3 http://172.16.0.228:31000/hostname; echo
k patch svc echo -p \
'{"spec":{"sessionAffinityConfig":{"clientIP":{"timeoutSeconds":60}}}}'
k get svc echo -o jsonpath='{.spec.sessionAffinityConfig}'; echo
k get endpointslices -l kubernetes.io/service-name=echo -o jsonpath=\
'{range .items[*].endpoints[*]}{.addresses[0]} {.targetRef.name} {.nodeName}{"\n"}{end}'
k get svc -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,\
ETP:.spec.externalTrafficPolicy,ITP:.spec.internalTrafficPolicyExpected (abridged):
echo-cc8555f69-kc99p
{"clientIP":{"timeoutSeconds":60}}
10.0.2.114 echo-cc8555f69-kc99p worker2
10.0.1.166 echo-cc8555f69-cvn8f worker1
NAME TYPE ETP ITP
docs ExternalName <none> <none>
echo ClusterIP <none> Cluster
echo-hl ClusterIP <none> Cluster
echo-lb LoadBalancer Cluster Cluster
echo-local ClusterIP <none> Local
echo-np NodePort Local Cluster
router ClusterIP <none> Cluster
web NodePort Cluster Cluster
If the first curl right after the patch returns nothing,
wait a second – Cilium needs a moment for the new entry.
Save your progress? Sign up for free
Checkpoint
k get svc echo-np -o jsonpath='{.spec.ports[0].nodePort} {.spec.externalTrafficPolicy}'Expected: 30080 Local
ssh [email protected] curl -s -m3 http://172.16.0.228:30080/clientipExpected: 172.16.0.123:<port> –
the real client IP.
k get svc echo-lbExpected: EXTERNAL-IP
<pending> (no LB IPAM pool until Day 15).
k exec client -- dig +short echo-hl.tag06.svc.cluster.localExpected: the two Pod IPs of echo, no
ClusterIP.
k exec client -- curl -s -m3 http://router/Expected: hallo vom router
Challenge
Set echo-lb to externalTrafficPolicy: Local
and show from router how an external load balancer
recognizes which nodes it may send traffic to.
Hint: With Local, Kubernetes assigns an
additional field spec.healthCheckNodePort to LoadBalancer
Services.
Solution
k patch svc echo-lb -p '{"spec":{"externalTrafficPolicy":"Local"}}'
HC=$(k get svc echo-lb -o jsonpath='{.spec.healthCheckNodePort}'); echo $HC
for n in 172.16.0.126 172.16.0.228 172.16.0.83; do
ssh [email protected] "curl -s -m3 -w ' %{http_code}' http://$n:$HC/"
echo " <- $n"
doneExpected:
31601
{"service":{"namespace":"tag06","name":"echo-lb"},"localEndpoints":0}
503 <- 172.16.0.126
{"service":{"namespace":"tag06","name":"echo-lb"},"localEndpoints":1}
200 <- 172.16.0.228
{"service":{"namespace":"tag06","name":"echo-lb"},"localEndpoints":1}
200 <- 172.16.0.83
Cilium answers the health check port itself (instead of kube-proxy):
200 with local backends, 503 without. Cloud load balancers check exactly
this port and take cp1 out of rotation. A NodePort Service
does not get the field – echo-np does not have it.
Save your progress? Sign up for free
Cleanup
On router, stop the test server with Ctrl+C, then:
rm -r ~/tag06
exitOn cp1:
k delete namespace tag06
k config set-context --current --namespace=defaultDay 6 changed nothing cluster-wide.
Exam Tip
k exposeknows--type,--port,--target-port,--cluster-ip=None, but no fixed NodePort and no policies – you set those withk patchor via a$dofile.externalTrafficPolicyonly affects NodePort/LoadBalancer,internalTrafficPolicyonly traffic from inside the cluster;Localwithout a local backend means: no answer.- Docs: kubernetes.io “Service”, “Service Internal Traffic Policy”, “Create an External Load Balancer” (section “Preserving the client source IP”).
Enterprise Note
externalTrafficPolicy: Localis common when backends need the real client IP (WAF, rate limits, audit) – then spread the Pods across the nodes (topology spread).- Instead of SNAT, Cilium can preserve the client IP even with
Clusterusing DSR (Direct Server Return); more on Day 7. - Session affinity by client IP is a stopgap; stateful apps are better
served by cookies at the gateway (Gateway API
sessionPersistence) or a shared session store. - Use ExternalName only for targets without TLS name verification or with a matching SAN – otherwise your own slice pointing to fixed IPs or an egress gateway (Day 16).
- Manually maintained EndpointSlices go stale easily; in production a controller maintains them, or a ServiceEntry-like solution in the mesh.