Skip to content
Hands-on IT IT Drills

Day 15: Exposing Networks

Goal: You assign LoadBalancer IPs from your own pool, announce them and the Pod networks via BGP to an external router and reach them from outside the cluster. · Duration: approx. 75 min · Prerequisite: Day 6

Briefing

  • CKNE: Advanced Traffic Management – “Implementing Routing to Expose Networks”.
  • LB-IPAM fills EXTERNAL-IP from a CiliumLoadBalancerIPPool. The IP only becomes reachable once someone tells the network where it lives: via BGP (route) or via L2 (ARP reply).
  • The pool 10.50.0.0/24 deliberately lies outside the home network 172.16.0.0/24 – it exists only through the BGP routes on the router.
  • ASN plan (private range 64512–65534): router = AS 65000, cluster ost (that is its name from Day 17 on; in the manifest just a freely chosen instance name) = AS 65010, every node is its own BGP speaker (eBGP).
            AS 65000                          AS 65010
 router .123 (FRR) ──eBGP── cp1 .126 · worker1 .228 · worker2 .83
   10.50.0.1/32 via all nodes (ECMP)         LB-IPAM 10.50.0.0/24
   10.0.N.0/24  via node N                   Pod CIDR per node

Drill 1: Situation – pending and no route

mkdir -p ~/ckne/tag15 && cd ~/ckne/tag15
k create ns tag15
k config set-context --current --namespace=tag15
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
  --replicas=2 --port=8080 -- /agnhost netexec --http-port=8080
k rollout status deployment echo
k expose deployment echo --name=echo-lb --type=LoadBalancer --port=80 \
  --target-port=8080
k get svc echo-lb
k get ippools
ssh [email protected] ip route get 10.50.0.1

Expected (abridged):

deployment "echo" successfully rolled out
service/echo-lb exposed
NAME      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
echo-lb   LoadBalancer   10.102.65.144   <pending>     80:31201/TCP   1s
No resources found
10.50.0.1 via 172.16.0.1 dev eth0 src 172.16.0.123 uid 1000

No pool, no IP. And today the router would send 10.50.0.0/24 to its default gateway – into the void.

Save your progress? Sign up for free

Drill 2: LB-IPAM pool

cat > pool.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: tag15-pool
spec:
  allowFirstLastIPs: "No"
  blocks:
  - cidr: 10.50.0.0/24
  serviceSelector:
    matchLabels:
      io.kubernetes.service.namespace: tag15
EOF
k apply -f pool.yaml
k get ippools
k get svc echo-lb
k get svc echo-lb -o jsonpath='{.status}'; echo
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 '10.50.0'

Expected (abridged):

ciliumloadbalancerippool.cilium.io/tag15-pool created
NAME         DISABLED   CONFLICTING   IPS AVAILABLE   AGE
tag15-pool   false      False         253             4s
NAME      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
echo-lb   LoadBalancer   10.102.65.144   10.50.0.1     80:31201/TCP   1m
{"conditions":[{...,"reason":"satisfied","status":"True",
"type":"cilium.io/IPAMRequestSatisfied"}],"loadBalancer":{"ingress":[{"ip":"10.50.0.1",
"ipMode":"VIP"}]}}
23   10.50.0.1:80/TCP        LoadBalancer   1 => 10.0.1.50:8080/TCP (active)
                                            2 => 10.0.1.132:8080/TCP (active)

Without allowFirstLastIPs: "No" the first Service gets 10.50.0.0 – technically valid, but some routers and firewalls dislike network and broadcast addresses. The serviceSelector with the automatic label io.kubernetes.service.namespace restricts the pool to tag15; a Service elsewhere stays <pending>:

k -n default create service loadbalancer fremd --tcp=80:80
k -n default get svc fremd
k -n default delete svc fremd

Expected (abridged):

NAME    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
fremd   LoadBalancer   10.99.116.203   <pending>     80:31368/TCP   3s

Save your progress? Sign up for free

Drill 3: FRR on the router

The router becomes a real BGP router. On Ubuntu the FRR package is enough; the BGP daemon is off by default:

ssh [email protected]
sudo apt-get install -y frr
sudo sed -i 's/^bgpd=no/bgpd=yes/' /etc/frr/daemons
sudo systemctl restart frr
sudo vtysh -c 'show version' | head -1
sudo vtysh

In vtysh (a Cisco-like shell), enter the neighbors:

conf t
router bgp 65000
 bgp router-id 172.16.0.123
 no bgp ebgp-requires-policy
 neighbor CILIUM peer-group
 neighbor CILIUM remote-as 65010
 neighbor 172.16.0.126 peer-group CILIUM
 neighbor 172.16.0.228 peer-group CILIUM
 neighbor 172.16.0.83 peer-group CILIUM
 address-family ipv4 unicast
  maximum-paths 8
end
write memory
show bgp summary
exit

Expected (abridged):

FRRouting 8.4.4 (prod) on Linux(6.8.0-149-generic).
Integrated configuration saved to /etc/frr/frr.conf
BGP router identifier 172.16.0.123, local AS number 65000 vrf-id 0
Neighbor        V         AS   MsgRcvd   MsgSent ...  Up/Down State/PfxRcd   PfxSnt
172.16.0.83     4      65010         0         0 ...    never         Idle        0
172.16.0.126    4      65010         0         0 ...    never         Idle        0
172.16.0.228    4      65010         0         0 ...    never         Idle        0

no bgp ebgp-requires-policy turns off the FRR default of accepting eBGP routes only with a route map – in the lab “accept everything” is enough. maximum-paths 8 enables ECMP: the same Service IP via several nodes. Leave vtysh with exit and the router with another exit – back on cp1.

Save your progress? Sign up for free

Drill 4: Enable the BGP Control Plane

helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
  --reuse-values --set bgpControlPlane.enabled=true
k -n kube-system rollout restart ds/cilium deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep enable-bgp-control-plane
k api-resources | grep -i bgp

Expected (abridged):

Release "cilium" has been upgraded. Happy Helming!
daemon set "cilium" successfully rolled out
enable-bgp-control-plane                          true
enable-bgp-control-plane-status-report            true
ciliumbgpadvertisements          cbgpadvert          cilium.io/v2   false   CiliumBGP...
ciliumbgpclusterconfigs          cbgpcluster         cilium.io/v2   false   CiliumBGP...
ciliumbgpnodeconfigoverrides     cbgpnodeoverride    cilium.io/v2   false   CiliumBGP...
ciliumbgpnodeconfigs             cbgpnode            cilium.io/v2   false   CiliumBGP...
ciliumbgppeerconfigs             cbgppeer            cilium.io/v2   false   CiliumBGP...

--reuse-values keeps all previous values (including those from other days). The ConfigMap changes immediately, but the agents only read it after the restart. In 1.20 the BGP API is cilium.io/v2; the old CiliumBGPPeeringPolicy no longer shows up.

Save your progress? Sign up for free

Drill 5: Establish peering

Three objects: the ClusterConfig selects nodes and neighbors, the PeerConfig holds per-neighbor settings (families, timers) and points via label to the Advertisements:

cat > bgp.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: tag15-bgp
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
  bgpInstances:
  - name: ost
    localASN: 65010
    peers:
    - name: router
      peerASN: 65000
      peerAddress: 172.16.0.123
      peerConfigRef:
        name: tag15-peer
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: tag15-peer
spec:
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: tag15
EOF
k apply -f bgp.yaml
cilium bgp peers
k get ciliumbgpnodeconfigs
ssh [email protected] "sudo vtysh -c 'show bgp summary'" | tail -5

Expected (abridged):

Node      Local AS   Peer AS   Peer Address   Session State   ...  Received   Advertised
cp1       65010      65000     172.16.0.123   established     ...  0          0
worker1   65010      65000     172.16.0.123   established     ...  0          0
worker2   65010      65000     172.16.0.123   established     ...  0          0
NAME      AGE
cp1       81s
worker1   81s
worker2   81s
Neighbor        V         AS   MsgRcvd   MsgSent ...  Up/Down State/PfxRcd   PfxSnt
172.16.0.83     4      65010         2         2 ... 00:00:16            0        0
172.16.0.126    4      65010         2         2 ... 00:00:16            0        0
172.16.0.228    4      65010         2         2 ... 00:00:16            0        0

The sessions are up, but Advertised 0: without a matching CiliumBGPAdvertisement, Cilium announces nothing. The operator creates the CiliumBGPNodeConfig objects per node from the ClusterConfig – their status shows the state of each session.

Save your progress? Sign up for free

Drill 6: Announce the Service IP

k label svc echo-lb bgp=on
cat > adv-svc.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: tag15-lb
  labels:
    advertise: tag15
spec:
  advertisements:
  - advertisementType: Service
    service:
      addresses:
      - LoadBalancerIP
    selector:
      matchLabels:
        bgp: "on"
EOF
k apply -f adv-svc.yaml
cilium bgp routes advertised ipv4 unicast

Expected (abridged):

Node      VRouter   Peer           Prefix         NextHop        Age   Attrs
cp1       65010     172.16.0.123   10.50.0.1/32   172.16.0.126   10s   [{Origin: i} ...
worker1   65010     172.16.0.123   10.50.0.1/32   172.16.0.228   10s   [{Origin: i} ...
worker2   65010     172.16.0.123   10.50.0.1/32   172.16.0.83    10s   [{Origin: i} ...

Now on the router:

ssh [email protected]
sudo vtysh -c 'show bgp ipv4 unicast'
ip route show 10.50.0.1
for i in 1 2 3 4; do curl -s -m3 http://10.50.0.1/hostname; echo; done
curl -s -m3 http://10.50.0.1/clientip; echo
exit

Expected (abridged):

   Network          Next Hop            Metric LocPrf Weight Path
*= 10.50.0.1/32     172.16.0.228                           0 65010 i
*>                  172.16.0.83                            0 65010 i
*=                  172.16.0.126                           0 65010 i
10.50.0.1 nhid 21 proto bgp metric 20
    nexthop via 172.16.0.83 dev eth0 weight 1
    nexthop via 172.16.0.126 dev eth0 weight 1
    nexthop via 172.16.0.228 dev eth0 weight 1
echo-cc8555f69-xgp7v
echo-cc8555f69-xgp7v
echo-cc8555f69-shgfk
echo-cc8555f69-shgfk
10.0.2.241:54950

*> is the best path, *= equal-cost multipath paths – the kernel distributes via ECMP across all three nodes. The client IP is cilium_host of worker2: with externalTrafficPolicy: Cluster the receiving node forwards with SNAT (Day 6).

Save your progress? Sign up for free

Drill 7: Announce the Pod CIDR and ETP Local

cat > adv-pod.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: tag15-podcidr
  labels:
    advertise: tag15
spec:
  advertisements:
  - advertisementType: PodCIDR
EOF
k apply -f adv-pod.yaml
k get pods -o wide
ssh [email protected] "ip route | grep '^10\.'"
P=$(k get pod -l app=echo -o jsonpath='{.items[0].status.podIP}'); echo $P
ssh [email protected] curl -s -m3 http://$P:8080/clientip; echo

Expected (abridged):

NAME                   READY   STATUS    ...   IP           NODE
echo-cc8555f69-shgfk   1/1     Running   ...   10.0.1.50    worker1
echo-cc8555f69-xgp7v   1/1     Running   ...   10.0.1.132   worker1
10.0.0.0/24 nhid 16 via 172.16.0.126 dev eth0 proto bgp metric 20
10.0.1.0/24 nhid 22 via 172.16.0.228 dev eth0 proto bgp metric 20
10.0.2.0/24 nhid 15 via 172.16.0.83 dev eth0 proto bgp metric 20
10.50.0.1 nhid 21 proto bgp metric 20
10.0.1.50
172.16.0.123:57256

Each node announces its own Pod CIDR from its CiliumNode; the router reaches Pods directly, without NAT – the Pod sees the real client IP. With externalTrafficPolicy: Local, Cilium withdraws the Service route from nodes without a backend:

k patch svc echo-lb -p '{"spec":{"externalTrafficPolicy":"Local"}}'
ssh [email protected] ip route show 10.50.0.1
ssh [email protected] curl -s -m3 http://10.50.0.1/clientip; echo

Expected:

service/echo-lb patched
10.50.0.1 nhid 22 via 172.16.0.228 dev eth0 proto bgp metric 20
172.16.0.123:43742

Only worker1 is left (both Pods run there) – that is the clean way to the real client IP without the detour via a deaf node (Day 6: curl exit 7).

Save your progress? Sign up for free

Drill 8: L2 Announcements – understand only

Without a BGP router it also works on Layer 2: Cilium answers ARP requests for LB IPs that lie in the node subnet, and one node per Service holds the IP via a lease.

cilium config view | grep enable-l2
k explain ciliuml2announcementpolicy.spec | grep -E '^  [a-z]'

Expected (abridged):

enable-l2-neigh-discovery                         false
externalIPs <boolean>
interfaces  <[]string>
loadBalancerIPs <boolean>
nodeSelector    <Object>
serviceSelector <Object>

A key enable-l2-announcements is missing from the configuration entirely – the feature is off, only the CRD is installed.

You would turn it on with l2announcements.enabled=true (plus higher k8sClientRateLimit values because of the leases) and a CiliumL2AnnouncementPolicy. In the lab it stays off: L2 needs a pool from 172.16.0.0/24 – but there the home router hands out addresses via DHCP. Cilium would answer ARP for IPs that might currently belong to a laptop and push it off the network. BGP with its own network 10.50.0.0/24 collides with nobody.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Create Service echo-fix (LoadBalancer, 80 → 8080, label bgp=on) and give it the fixed IP 10.50.0.50. Target time: 90 s
  2. Set hold time 9 s and keepalive 3 s for all peers of the PeerConfig and check it on the router. Target time: 90 s
  3. Show which prefixes worker1 announces to the router. Target time: 30 s

Solution

k expose deployment echo --name=echo-fix --type=LoadBalancer --port=80 \
  --target-port=8080 --labels=bgp=on
k annotate svc echo-fix lbipam.cilium.io/ips=10.50.0.50
k get svc echo-fix
ssh [email protected] curl -s -m3 http://10.50.0.50/hostname; echo
k patch ciliumbgppeerconfig tag15-peer --type=merge \
  -p '{"spec":{"timers":{"holdTimeSeconds":9,"keepAliveTimeSeconds":3}}}'
cilium bgp peers
ssh [email protected] "sudo vtysh -c 'show bgp neighbors 172.16.0.228'" \
  | grep -iE 'hold time|BGP state'
cilium bgp routes advertised ipv4 unicast --node worker1

Expected (abridged):

NAME       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
echo-fix   LoadBalancer   10.101.250.213   10.50.0.50    80:31309/TCP   5s
echo-cc8555f69-xgp7v
ciliumbgppeerconfig.cilium.io/tag15-peer patched
Node      Local AS   Peer AS   Peer Address   Session State   Uptime   ...
cp1       65010      65000     172.16.0.123   established     5s       ...
  BGP state = Established, up for 00:00:06
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
worker1   65010     172.16.0.123   10.0.1.0/24    172.16.0.228   ...
          65010     172.16.0.123   10.50.0.1/32   172.16.0.228   ...
          65010     172.16.0.123   10.50.0.50/32  172.16.0.228   ...

The annotation moves the already assigned IP immediately. New timers re-establish the session (uptime 5 s) – in production a brief route change. The smaller value of both sides is negotiated, which is why FRR reports 9 s instead of its own 180 s.

Save your progress? Sign up for free

Checkpoint

cilium bgp peers

Expected: three lines established to peer 172.16.0.123.

k get svc echo-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}'; echo

Expected: 10.50.0.1

ssh [email protected] curl -s -m3 http://10.50.0.1/hostname

Expected: a Pod name such as echo-cc8555f69-xgp7v.

ssh [email protected] "ip route | grep proto.bgp"

Expected: three Pod CIDRs 10.0.N.0/24 and the Service IPs 10.50.0.x with proto bgp.

helm get values cilium -n kube-system | grep -A1 bgpControlPlane

Expected: enabled: true – stays after the cleanup for Day 21 (Round 7) and Appendix B (Task 10), as does FRR on the router.

Challenge

Additionally announce the ClusterIP of Service echo (create it: port 80 → 8080, label bgp=clusterip) and call it from the router. Explain what happens.

Hint: The route arrives – whether a node accepts the packets is decided by a Helm value under bpf..

Solution

k expose deployment echo --port=80 --target-port=8080 --labels=bgp=clusterip
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: tag15-clusterip
  labels:
    advertise: tag15
spec:
  advertisements:
  - advertisementType: Service
    service:
      addresses:
      - ClusterIP
    selector:
      matchLabels:
        bgp: clusterip
EOF
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}'); echo $CIP
ssh [email protected] ip route show $CIP
ssh [email protected] curl -s -m3 http://$CIP/hostname; echo "exit=$?"
hubble observe -P --port-forward-port 0 --ip $CIP --last 3
cilium config view | grep external-clusterip

Expected (abridged):

10.105.95.126
10.105.95.126 nhid 21 proto bgp metric 20
    nexthop via 172.16.0.83 dev eth0 weight 1
    ...
exit=28
Nov 18 23:44:45.943: 172.16.0.123:36726 (world) <> tag15/echo:80 (world) Is a
ClusterIP DROPPED (TCP Flags: SYN)
bpf-lb-external-clusterip                         false

The route is correct, but Cilium drops packets from outside to a ClusterIP (drop reason “Is a ClusterIP”). Only bpf.lbExternalClusterIP=true opens ClusterIPs to cluster-external clients – not set in the lab; for external clients the LoadBalancer Service is the right way.

Save your progress? Sign up for free

Cleanup

k delete ciliumbgpadvertisement tag15-lb tag15-podcidr tag15-clusterip \
  --ignore-not-found
k delete -f bgp.yaml
k delete -f pool.yaml
k delete namespace tag15
k config set-context --current --namespace=default
ssh [email protected] "ip route | grep -c proto.bgp"

Expected: 0 – without advertisements and sessions all BGP routes are gone.

bgpControlPlane.enabled=true and FRR on the router (AS 65000, neighbors stay configured, status Active) remain in place for the Competition (Day 21, Round 7) and the practice exam (Appendix B, Task 10) – both are additive and do not disturb other days.

Exam Tip

  • Check the chain from top to bottom: Service has an IP (k get svc) → advertisement matches (label of the PeerConfig and Service selector) → cilium bgp peers established and Advertised > 0 → route on the router.
  • Advertised 0 with an established session almost always means: the PeerConfig’s label selector does not match any CiliumBGPAdvertisement.
  • Docs: docs.cilium.io “BGP Control Plane” (section “BGP Configuration Resources”), “LoadBalancer IP Address Management (LB IPAM)”, “L2 Announcements”.

Enterprise Note

  • In the data center, nodes peer with the top-of-rack switches (often one AS per rack, nodeSelector per rack label); BFD and Graceful Restart shorten downtime.
  • Set prefix filters on the router side (route maps, max-prefix) – otherwise a misconfigured cluster can announce arbitrary networks.
  • Secure sessions with a password (authSecretRef, TCP-MD5).
  • Separate pools per tenant or environment via serviceSelector; the CONFLICTING column shows collisions.
  • Use L2 Announcements only in dedicated subnets with reserved address ranges (excluded from DHCP).