Day 15: Exposing Networks
Goal: You assign LoadBalancer IPs from your own pool, announce them and the Pod networks via BGP to an external router and reach them from outside the cluster. · Duration: approx. 75 min · Prerequisite: Day 6
Briefing
- CKNE: Advanced Traffic Management – “Implementing Routing to Expose Networks”.
- LB-IPAM fills
EXTERNAL-IPfrom aCiliumLoadBalancerIPPool. The IP only becomes reachable once someone tells the network where it lives: via BGP (route) or via L2 (ARP reply). - The pool 10.50.0.0/24 deliberately lies outside the
home network 172.16.0.0/24 – it exists only through the BGP routes on
the
router. - ASN plan (private range 64512–65534):
router= AS 65000, clusterost(that is its name from Day 17 on; in the manifest just a freely chosen instance name) = AS 65010, every node is its own BGP speaker (eBGP).
AS 65000 AS 65010
router .123 (FRR) ──eBGP── cp1 .126 · worker1 .228 · worker2 .83
10.50.0.1/32 via all nodes (ECMP) LB-IPAM 10.50.0.0/24
10.0.N.0/24 via node N Pod CIDR per node
Drill 1: Situation – pending and no route
mkdir -p ~/ckne/tag15 && cd ~/ckne/tag15
k create ns tag15
k config set-context --current --namespace=tag15
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
--replicas=2 --port=8080 -- /agnhost netexec --http-port=8080
k rollout status deployment echo
k expose deployment echo --name=echo-lb --type=LoadBalancer --port=80 \
--target-port=8080
k get svc echo-lb
k get ippools
ssh [email protected] ip route get 10.50.0.1Expected (abridged):
deployment "echo" successfully rolled out
service/echo-lb exposed
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-lb LoadBalancer 10.102.65.144 <pending> 80:31201/TCP 1s
No resources found
10.50.0.1 via 172.16.0.1 dev eth0 src 172.16.0.123 uid 1000
No pool, no IP. And today the router would send
10.50.0.0/24 to its default gateway – into the void.
Save your progress? Sign up for free
Drill 2: LB-IPAM pool
cat > pool.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
name: tag15-pool
spec:
allowFirstLastIPs: "No"
blocks:
- cidr: 10.50.0.0/24
serviceSelector:
matchLabels:
io.kubernetes.service.namespace: tag15
EOF
k apply -f pool.yaml
k get ippools
k get svc echo-lb
k get svc echo-lb -o jsonpath='{.status}'; echo
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 '10.50.0'Expected (abridged):
ciliumloadbalancerippool.cilium.io/tag15-pool created
NAME DISABLED CONFLICTING IPS AVAILABLE AGE
tag15-pool false False 253 4s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-lb LoadBalancer 10.102.65.144 10.50.0.1 80:31201/TCP 1m
{"conditions":[{...,"reason":"satisfied","status":"True",
"type":"cilium.io/IPAMRequestSatisfied"}],"loadBalancer":{"ingress":[{"ip":"10.50.0.1",
"ipMode":"VIP"}]}}
23 10.50.0.1:80/TCP LoadBalancer 1 => 10.0.1.50:8080/TCP (active)
2 => 10.0.1.132:8080/TCP (active)
Without allowFirstLastIPs: "No" the first Service gets
10.50.0.0 – technically valid, but some routers and
firewalls dislike network and broadcast addresses. The
serviceSelector with the automatic label
io.kubernetes.service.namespace restricts the pool to
tag15; a Service elsewhere stays
<pending>:
k -n default create service loadbalancer fremd --tcp=80:80
k -n default get svc fremd
k -n default delete svc fremdExpected (abridged):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fremd LoadBalancer 10.99.116.203 <pending> 80:31368/TCP 3s
Save your progress? Sign up for free
Drill 3: FRR on the router
The router becomes a real BGP router. On Ubuntu the FRR
package is enough; the BGP daemon is off by default:
ssh [email protected]
sudo apt-get install -y frr
sudo sed -i 's/^bgpd=no/bgpd=yes/' /etc/frr/daemons
sudo systemctl restart frr
sudo vtysh -c 'show version' | head -1
sudo vtyshIn vtysh (a Cisco-like shell), enter the neighbors:
conf t
router bgp 65000
bgp router-id 172.16.0.123
no bgp ebgp-requires-policy
neighbor CILIUM peer-group
neighbor CILIUM remote-as 65010
neighbor 172.16.0.126 peer-group CILIUM
neighbor 172.16.0.228 peer-group CILIUM
neighbor 172.16.0.83 peer-group CILIUM
address-family ipv4 unicast
maximum-paths 8
end
write memory
show bgp summary
exit
Expected (abridged):
FRRouting 8.4.4 (prod) on Linux(6.8.0-149-generic).
Integrated configuration saved to /etc/frr/frr.conf
BGP router identifier 172.16.0.123, local AS number 65000 vrf-id 0
Neighbor V AS MsgRcvd MsgSent ... Up/Down State/PfxRcd PfxSnt
172.16.0.83 4 65010 0 0 ... never Idle 0
172.16.0.126 4 65010 0 0 ... never Idle 0
172.16.0.228 4 65010 0 0 ... never Idle 0
no bgp ebgp-requires-policy turns off the FRR default of
accepting eBGP routes only with a route map – in the lab “accept
everything” is enough. maximum-paths 8 enables ECMP: the
same Service IP via several nodes. Leave vtysh with exit
and the router with another exit – back on
cp1.
Save your progress? Sign up for free
Drill 4: Enable the BGP Control Plane
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
--reuse-values --set bgpControlPlane.enabled=true
k -n kube-system rollout restart ds/cilium deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep enable-bgp-control-plane
k api-resources | grep -i bgpExpected (abridged):
Release "cilium" has been upgraded. Happy Helming!
daemon set "cilium" successfully rolled out
enable-bgp-control-plane true
enable-bgp-control-plane-status-report true
ciliumbgpadvertisements cbgpadvert cilium.io/v2 false CiliumBGP...
ciliumbgpclusterconfigs cbgpcluster cilium.io/v2 false CiliumBGP...
ciliumbgpnodeconfigoverrides cbgpnodeoverride cilium.io/v2 false CiliumBGP...
ciliumbgpnodeconfigs cbgpnode cilium.io/v2 false CiliumBGP...
ciliumbgppeerconfigs cbgppeer cilium.io/v2 false CiliumBGP...
--reuse-values keeps all previous values (including
those from other days). The ConfigMap changes immediately, but the
agents only read it after the restart. In 1.20 the BGP API is
cilium.io/v2; the old CiliumBGPPeeringPolicy
no longer shows up.
Save your progress? Sign up for free
Drill 5: Establish peering
Three objects: the ClusterConfig selects nodes and neighbors, the PeerConfig holds per-neighbor settings (families, timers) and points via label to the Advertisements:
cat > bgp.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
name: tag15-bgp
spec:
nodeSelector:
matchLabels:
kubernetes.io/os: linux
bgpInstances:
- name: ost
localASN: 65010
peers:
- name: router
peerASN: 65000
peerAddress: 172.16.0.123
peerConfigRef:
name: tag15-peer
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
name: tag15-peer
spec:
families:
- afi: ipv4
safi: unicast
advertisements:
matchLabels:
advertise: tag15
EOF
k apply -f bgp.yaml
cilium bgp peers
k get ciliumbgpnodeconfigs
ssh [email protected] "sudo vtysh -c 'show bgp summary'" | tail -5Expected (abridged):
Node Local AS Peer AS Peer Address Session State ... Received Advertised
cp1 65010 65000 172.16.0.123 established ... 0 0
worker1 65010 65000 172.16.0.123 established ... 0 0
worker2 65010 65000 172.16.0.123 established ... 0 0
NAME AGE
cp1 81s
worker1 81s
worker2 81s
Neighbor V AS MsgRcvd MsgSent ... Up/Down State/PfxRcd PfxSnt
172.16.0.83 4 65010 2 2 ... 00:00:16 0 0
172.16.0.126 4 65010 2 2 ... 00:00:16 0 0
172.16.0.228 4 65010 2 2 ... 00:00:16 0 0
The sessions are up, but Advertised 0: without a
matching CiliumBGPAdvertisement, Cilium announces nothing.
The operator creates the CiliumBGPNodeConfig objects per
node from the ClusterConfig – their status shows the state
of each session.
Save your progress? Sign up for free
Drill 6: Announce the Service IP
k label svc echo-lb bgp=on
cat > adv-svc.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
name: tag15-lb
labels:
advertise: tag15
spec:
advertisements:
- advertisementType: Service
service:
addresses:
- LoadBalancerIP
selector:
matchLabels:
bgp: "on"
EOF
k apply -f adv-svc.yaml
cilium bgp routes advertised ipv4 unicastExpected (abridged):
Node VRouter Peer Prefix NextHop Age Attrs
cp1 65010 172.16.0.123 10.50.0.1/32 172.16.0.126 10s [{Origin: i} ...
worker1 65010 172.16.0.123 10.50.0.1/32 172.16.0.228 10s [{Origin: i} ...
worker2 65010 172.16.0.123 10.50.0.1/32 172.16.0.83 10s [{Origin: i} ...
Now on the router:
ssh [email protected]
sudo vtysh -c 'show bgp ipv4 unicast'
ip route show 10.50.0.1
for i in 1 2 3 4; do curl -s -m3 http://10.50.0.1/hostname; echo; done
curl -s -m3 http://10.50.0.1/clientip; echo
exitExpected (abridged):
Network Next Hop Metric LocPrf Weight Path
*= 10.50.0.1/32 172.16.0.228 0 65010 i
*> 172.16.0.83 0 65010 i
*= 172.16.0.126 0 65010 i
10.50.0.1 nhid 21 proto bgp metric 20
nexthop via 172.16.0.83 dev eth0 weight 1
nexthop via 172.16.0.126 dev eth0 weight 1
nexthop via 172.16.0.228 dev eth0 weight 1
echo-cc8555f69-xgp7v
echo-cc8555f69-xgp7v
echo-cc8555f69-shgfk
echo-cc8555f69-shgfk
10.0.2.241:54950
*> is the best path, *= equal-cost
multipath paths – the kernel distributes via ECMP across all three
nodes. The client IP is cilium_host of worker2: with
externalTrafficPolicy: Cluster the receiving node forwards
with SNAT (Day 6).
Save your progress? Sign up for free
Drill 7: Announce the Pod CIDR and ETP Local
cat > adv-pod.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
name: tag15-podcidr
labels:
advertise: tag15
spec:
advertisements:
- advertisementType: PodCIDR
EOF
k apply -f adv-pod.yaml
k get pods -o wide
ssh [email protected] "ip route | grep '^10\.'"
P=$(k get pod -l app=echo -o jsonpath='{.items[0].status.podIP}'); echo $P
ssh [email protected] curl -s -m3 http://$P:8080/clientip; echoExpected (abridged):
NAME READY STATUS ... IP NODE
echo-cc8555f69-shgfk 1/1 Running ... 10.0.1.50 worker1
echo-cc8555f69-xgp7v 1/1 Running ... 10.0.1.132 worker1
10.0.0.0/24 nhid 16 via 172.16.0.126 dev eth0 proto bgp metric 20
10.0.1.0/24 nhid 22 via 172.16.0.228 dev eth0 proto bgp metric 20
10.0.2.0/24 nhid 15 via 172.16.0.83 dev eth0 proto bgp metric 20
10.50.0.1 nhid 21 proto bgp metric 20
10.0.1.50
172.16.0.123:57256
Each node announces its own Pod CIDR from its CiliumNode; the
router reaches Pods directly, without NAT – the Pod sees
the real client IP. With externalTrafficPolicy: Local,
Cilium withdraws the Service route from nodes without a backend:
k patch svc echo-lb -p '{"spec":{"externalTrafficPolicy":"Local"}}'
ssh [email protected] ip route show 10.50.0.1
ssh [email protected] curl -s -m3 http://10.50.0.1/clientip; echoExpected:
service/echo-lb patched
10.50.0.1 nhid 22 via 172.16.0.228 dev eth0 proto bgp metric 20
172.16.0.123:43742
Only worker1 is left (both Pods run there) – that is the clean way to the real client IP without the detour via a deaf node (Day 6: curl exit 7).
Save your progress? Sign up for free
Drill 8: L2 Announcements – understand only
Without a BGP router it also works on Layer 2: Cilium answers ARP requests for LB IPs that lie in the node subnet, and one node per Service holds the IP via a lease.
cilium config view | grep enable-l2
k explain ciliuml2announcementpolicy.spec | grep -E '^ [a-z]'Expected (abridged):
enable-l2-neigh-discovery false
externalIPs <boolean>
interfaces <[]string>
loadBalancerIPs <boolean>
nodeSelector <Object>
serviceSelector <Object>
A key enable-l2-announcements is missing from the
configuration entirely – the feature is off, only the CRD is
installed.
You would turn it on with l2announcements.enabled=true
(plus higher k8sClientRateLimit values because of the
leases) and a CiliumL2AnnouncementPolicy. In the
lab it stays off: L2 needs a pool from 172.16.0.0/24 – but
there the home router hands out addresses via DHCP. Cilium would answer
ARP for IPs that might currently belong to a laptop and push it off the
network. BGP with its own network 10.50.0.0/24 collides with nobody.
Save your progress? Sign up for free
Exercise: Speed Drills
- Create Service
echo-fix(LoadBalancer, 80 → 8080, labelbgp=on) and give it the fixed IP 10.50.0.50. Target time: 90 s - Set hold time 9 s and keepalive 3 s for all peers of the PeerConfig
and check it on the
router. Target time: 90 s - Show which prefixes worker1 announces to the
router. Target time: 30 s
Solution
k expose deployment echo --name=echo-fix --type=LoadBalancer --port=80 \
--target-port=8080 --labels=bgp=on
k annotate svc echo-fix lbipam.cilium.io/ips=10.50.0.50
k get svc echo-fix
ssh [email protected] curl -s -m3 http://10.50.0.50/hostname; echo
k patch ciliumbgppeerconfig tag15-peer --type=merge \
-p '{"spec":{"timers":{"holdTimeSeconds":9,"keepAliveTimeSeconds":3}}}'
cilium bgp peers
ssh [email protected] "sudo vtysh -c 'show bgp neighbors 172.16.0.228'" \
| grep -iE 'hold time|BGP state'
cilium bgp routes advertised ipv4 unicast --node worker1Expected (abridged):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-fix LoadBalancer 10.101.250.213 10.50.0.50 80:31309/TCP 5s
echo-cc8555f69-xgp7v
ciliumbgppeerconfig.cilium.io/tag15-peer patched
Node Local AS Peer AS Peer Address Session State Uptime ...
cp1 65010 65000 172.16.0.123 established 5s ...
BGP state = Established, up for 00:00:06
Hold time is 9 seconds, keepalive interval is 3 seconds
Configured hold time is 180 seconds, keepalive interval is 60 seconds
worker1 65010 172.16.0.123 10.0.1.0/24 172.16.0.228 ...
65010 172.16.0.123 10.50.0.1/32 172.16.0.228 ...
65010 172.16.0.123 10.50.0.50/32 172.16.0.228 ...
The annotation moves the already assigned IP immediately. New timers re-establish the session (uptime 5 s) – in production a brief route change. The smaller value of both sides is negotiated, which is why FRR reports 9 s instead of its own 180 s.
Save your progress? Sign up for free
Checkpoint
cilium bgp peersExpected: three lines established to
peer 172.16.0.123.
k get svc echo-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}'; echoExpected: 10.50.0.1
ssh [email protected] curl -s -m3 http://10.50.0.1/hostnameExpected: a Pod name such as
echo-cc8555f69-xgp7v.
ssh [email protected] "ip route | grep proto.bgp"Expected: three Pod CIDRs 10.0.N.0/24
and the Service IPs 10.50.0.x with
proto bgp.
helm get values cilium -n kube-system | grep -A1 bgpControlPlaneExpected: enabled: true – stays after
the cleanup for Day 21 (Round 7) and Appendix B (Task 10), as does FRR
on the router.
Challenge
Additionally announce the ClusterIP of Service echo
(create it: port 80 → 8080, label bgp=clusterip) and call
it from the router. Explain what happens.
Hint: The route arrives – whether a node accepts the
packets is decided by a Helm value under bpf..
Solution
k expose deployment echo --port=80 --target-port=8080 --labels=bgp=clusterip
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
name: tag15-clusterip
labels:
advertise: tag15
spec:
advertisements:
- advertisementType: Service
service:
addresses:
- ClusterIP
selector:
matchLabels:
bgp: clusterip
EOF
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}'); echo $CIP
ssh [email protected] ip route show $CIP
ssh [email protected] curl -s -m3 http://$CIP/hostname; echo "exit=$?"
hubble observe -P --port-forward-port 0 --ip $CIP --last 3
cilium config view | grep external-clusteripExpected (abridged):
10.105.95.126
10.105.95.126 nhid 21 proto bgp metric 20
nexthop via 172.16.0.83 dev eth0 weight 1
...
exit=28
Nov 18 23:44:45.943: 172.16.0.123:36726 (world) <> tag15/echo:80 (world) Is a
ClusterIP DROPPED (TCP Flags: SYN)
bpf-lb-external-clusterip false
The route is correct, but Cilium drops packets from outside to a
ClusterIP (drop reason “Is a ClusterIP”). Only
bpf.lbExternalClusterIP=true opens ClusterIPs to
cluster-external clients – not set in the lab; for external clients the
LoadBalancer Service is the right way.
Save your progress? Sign up for free
Cleanup
k delete ciliumbgpadvertisement tag15-lb tag15-podcidr tag15-clusterip \
--ignore-not-found
k delete -f bgp.yaml
k delete -f pool.yaml
k delete namespace tag15
k config set-context --current --namespace=default
ssh [email protected] "ip route | grep -c proto.bgp"Expected: 0 – without advertisements
and sessions all BGP routes are gone.
bgpControlPlane.enabled=true and FRR on the
router (AS 65000, neighbors stay configured, status
Active) remain in place for the Competition (Day 21, Round
7) and the practice exam (Appendix B, Task 10) – both are additive and
do not disturb other days.
Exam Tip
- Check the chain from top to bottom: Service has an IP
(
k get svc) → advertisement matches (label of the PeerConfig and Service selector) →cilium bgp peersestablished andAdvertised > 0→ route on the router. Advertised 0with an established session almost always means: the PeerConfig’s label selector does not match anyCiliumBGPAdvertisement.- Docs: docs.cilium.io “BGP Control Plane” (section “BGP Configuration Resources”), “LoadBalancer IP Address Management (LB IPAM)”, “L2 Announcements”.
Enterprise Note
- In the data center, nodes peer with the top-of-rack switches (often
one AS per rack,
nodeSelectorper rack label); BFD and Graceful Restart shorten downtime. - Set prefix filters on the router side (route maps, max-prefix) – otherwise a misconfigured cluster can announce arbitrary networks.
- Secure sessions with a password (
authSecretRef, TCP-MD5). - Separate pools per tenant or environment via
serviceSelector; theCONFLICTINGcolumn shows collisions. - Use L2 Announcements only in dedicated subnets with reserved address ranges (excluded from DHCP).