Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

The most important commands and snippets from all days, ordered by curriculum domain. The days are in parentheses. You replace the placeholders in angle brackets; IPs are those of the lab (cp1 172.16.0.126, worker1 .228, worker2 .83, router .123, west .182).

Exam Craft (Days 0, 1)

source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k
export do="--dry-run=client -o yaml"         # generate YAML instead of creating
export now="--force --grace-period=0"        # delete immediately
source <(cilium completion bash)
source <(hubble completion bash)
k config use-context <context>                # first action of every task
k config set-context --current --namespace=<ns>
k config view --minify -o jsonpath='{..namespace}'; echo
ssh [email protected]                        # node work, exit at the end

Always run cilium-dbg in the agent of the affected node – ds/cilium hits a random one:

CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=<node> -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --brief
N=$(k get pod <pod> -o jsonpath='{.spec.nodeName}')   # node of a Pod
hubble status -P --port-forward-port 0        # -P: port-forward to Relay per call

Linux tools (Days 1, 4)

ip -br link
ip -br addr show cilium_host
ip route
ip route get 10.0.2.248                       # which route does this IP take?
sudo ss -tlnp '( sport = :6443 or sport = :10250 or sport = :4240 or sport = :4244 )'
sudo iptables-save | grep -c KUBE-SVC         # 0 = no kube-proxy
sudo nft list tables
sudo tcpdump -i any -nn -c 4 host 10.0.2.248 and tcp port 80
sudo tcpdump -l -i eth0 -nn udp port 8472     # VXLAN between nodes
sudo tcpdump -nn -e -i eth0 -c 2 tcp port 30707   # -e shows MAC addresses (DSR)

Into a Pod’s network (on the Pod’s node):

sudo crictl pods --namespace <ns> --name <pod>
ID=$(sudo crictl ps --name <container> -q --label io.kubernetes.pod.namespace=<ns>)
P=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
sudo nsenter -t $P -n ip -br addr
sudo nsenter -t $P -n ip route
sudo nsenter -t $P -n ss -tlnp
sudo nsenter -t $P -n tcpdump -i eth0 -nn -c 2 icmp
ip -o link | awk -F': ' '$1==33{print $2}'    # host end for eth0@if33

kubectl debug with netshoot, without SSH:

k debug <pod> -it --image=nicolaka/netshoot:v0.16 --target=<container> \
  --profile=netadmin                          # ephemeral container in the Pod netns
k debug node/worker2 -it --image=nicolaka/netshoot:v0.16 --profile=netadmin
# afterwards delete the Pod node-debugger-worker2-…, it stays behind
k run c1 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity

Core Infrastructure and CNI (Days 2–5)

CNI on the node (Day 2):

sudo ls -la /etc/cni/net.d                    # drwx------, only with sudo
sudo cat /etc/cni/net.d/05-cilium.conflist
ls /opt/cni/bin
sudo crictl info | jq '.cniconfig | {PluginConfDir, PluginDirs, PluginMaxConfNum}'
sudo journalctl -u containerd --since -1min | grep -m1 'reload cni'
cilium config view | grep -E '^cni-|write-cni'
k -n kube-system get cm cilium-config -o jsonpath='{.data.cni-exclusive}'; echo
P=$(sudo crictl pods --name <pod> --namespace <ns> -q)
sudo crictl inspectp $P | jq -c '.info.cniResult.Interfaces.eth0.IPConfigs'
  • If 05-cilium.conflist is missing, the agent rewrites it (Wrote CNI in the log).
  • With cni.exclusive=true, Cilium renames foreign files to *.cilium_bak.

Changing Helm values (pattern for every day)

helm get values cilium -n kube-system -o yaml > values-vorher.yaml   # back up first
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
  --reuse-values --set <key>=<value>          # without --reuse-values: all reset
k -n kube-system rollout restart ds/cilium    # for operator values also
k -n kube-system rollout restart deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep <key>               # result in the ConfigMap
helm get values cilium -n kube-system --all | grep -A13 '^cni:'   # incl. defaults
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
  -f values-vorher.yaml                       # way back
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
  • cilium config set debug true --restart=false only changes the ConfigMap – drift that Helm 4 reports as a conflict on rollback (helm rollback … --force-conflicts).
  • Values from a file: helm upgrade … --reuse-values -f datei.yaml.
Helm value Effect Day
debug.enabled=true debug logs in the agent 2
ipam.operator.clusterPoolIPv4PodCIDRList Pod pool, only extend 3
cni.exclusive=false tolerate foreign CNI files (Multus) 5
routingMode=native, autoDirectNodeRoutes=true native routing 7
ipv4NativeRoutingCIDR=10.0.0.0/16 network without masquerading 7
loadBalancer.mode=dsr, loadBalancer.algorithm=maglev DSR, Maglev 7
gatewayAPI.enabled=true Gateway controller 10
encryption.enabled=true, encryption.type=wireguard WireGuard 13
encryption.nodeEncryption=true node-to-node too 13
authentication.mutual.spire.enabled=true mutual auth with SPIRE 14
bgpControlPlane.enabled=true BGP Control Plane 15
egressGateway.enabled=true, bpf.masquerade=true Egress Gateway 16
cluster.name=ost, cluster.id=1 prerequisite for ClusterMesh 17
clustermesh.mcsapi.enabled=true MCS API 17
hubble.export.dynamic.enabled=true flow logs to a file 20

Values per node (Day 21, round 1) – CiliumNodeConfig overrides the ConfigMap:

k get ciliumnodeconfigs -A                    # spec.nodeSelector + spec.defaults
k -n kube-system logs $CP -c config | grep 'Got configuration'   # sources read
  • cilium config view only reads the ConfigMap and does not show the override.

IPAM (Day 3)

cilium config view | grep -E '^ipam |cluster-pool'
k get ciliumnodes -o custom-columns='NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs'
k get ciliumnodes -o jsonpath=\
'{range .items[*]}{.metadata.name}{"\t"}{.status.ipam.operator-status}{"\n"}{end}'
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --all-addresses \
  | sed -n '/^IPAM/,/^IPv4 BIG/p'
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache get <ip>
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg ip get -v <ip>/32
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.0.0.0/16,10.2.0.0/16}'
k -n kube-system rollout restart deployment cilium-operator
  • node.spec.podCIDR is only filled in IPAM mode kubernetes – with cluster-pool, the CiliumNode is what counts.
  • CiliumPodIPPool plus the annotation ipam.cilium.io/ip-pool only takes effect with ipam.mode=multi-pool (created in the lab, but without effect).

Debugging Pod connectivity (Day 4)

Order: Pod Ready? → Pod IP directly → EndpointSlices → DNS in the Pod → drops.

k get pods -o wide
k exec c1 -- curl -s -o /dev/null -w '%{remote_ip} %{http_code}\n' http://<pod-ip>
k exec c1 -- traceroute -n -w 1 -q 1 <pod-ip>
k get endpointslices
k describe svc <svc> | grep -E 'Selector|Endpoints'
k get pods --show-labels
k exec c1 -- cat /etc/resolv.conf
k exec c1 -- nslookup web
hubble observe -P --port-forward-port 0 --pod <ns>/c1 --tcp-flags SYN --last 2
cilium config view | grep -E '^routing-mode|^tunnel-protocol|^auto-direct'
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg monitor --related-to <ep-id>
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg identity get <id>
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg node list
cilium connectivity test --test-namespace tag04-ct --test no-policies
k set selector svc web-sel app=web                                        # repair
k patch svc api -p '{"spec":{"ports":[{"port":80,"targetPort":"web"}]}}'  # port name
  • No flow in Hubble = no backend; RST = wrong port; 0/1 Ready = probe.

Multus (Day 5)

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-lab
spec:
  config: |
    {"cniVersion": "0.3.1", "type": "macvlan", "master": "eth0", "mode": "bridge",
     "capabilities": {"ips": true}, "ipam": {"type": "static"}}
k get net-attach-def -A
NET1='[{"name":"macvlan-lab","ips":["192.168.77.11/24"]}]'
k run m1 --image=nicolaka/netshoot:v0.16 \
  --annotations=k8s.v1.cni.cncf.io/networks="$NET1" -- sleep infinity
k exec m1 -- ip -br addr                      # eth0 = Cilium, net1 = Multus
k get pod m1 \
  -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'; echo
  • JSON form of the annotation: name, interface, ips; short form only the name. 00-multus.conf must be the first file in /etc/cni/net.d. net1 bypasses Cilium: no policy, no Hubble.

Service Networking and DNS (Days 6–9, 17)

Services at L4 (Day 6):

k expose deployment echo --port=80 --target-port=8080
k expose deployment echo --name=echo-np --type=NodePort --port=80 --target-port=8080
k patch svc echo-np -p '{"spec":{"ports":[{"port":80,"nodePort":30080}]}}'
k expose deployment echo --name=echo-lb --type=LoadBalancer --port=80 \
  --target-port=8080
k expose deployment echo --name=echo-hl --cluster-ip=None --port=8080
k create service externalname docs --external-name=kubernetes.io
k patch svc echo -p '{"spec":{"sessionAffinity":"ClientIP"}}'
k patch svc echo-np -p '{"spec":{"externalTrafficPolicy":"Local"}}'
k patch svc echo -p '{"spec":{"internalTrafficPolicy":"Local"}}'
k get svc echo-lb -o jsonpath='{.spec.healthCheckNodePort}'   # with ETP Local

Service without a selector, with its own EndpointSlice:

k create service clusterip router --tcp=80:8000 $do > router-svc.yaml
sed -i '/selector:/,+1d' router-svc.yaml
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: router-1
  labels:
    kubernetes.io/service-name: router
    endpointslice.kubernetes.io/managed-by: tag06.example.com
addressType: IPv4
ports:
- name: 80-8000                               # same name as in the Service port
  protocol: TCP
  port: 8000
endpoints:
- addresses:
  - 172.16.0.123
  • externalTrafficPolicy only affects NodePort/LoadBalancer, internalTrafficPolicy only traffic from inside the cluster; Local without a local backend = no answer.

kube-proxy and Cilium KPR (Days 0, 7)

k -n kube-system get cm kube-proxy            # NotFound = no kube-proxy
k get pods -A -l k8s-app=kube-proxy
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep KubeProxyReplacement
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --verbose \
  | sed -n '/KubeProxyReplacement Details/,/HostPort/p'
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}')
X="kubectl -n kube-system exec $CP -c cilium-agent --"   # kubectl, not the alias
$X cilium-dbg service list | grep -A1 "$CIP:"
$X cilium-dbg bpf lb list | grep "$CIP:80"
$X cilium-dbg map list | grep -E '^Name|lb4_'
$X cilium-dbg bpf lb maglev list | cut -c1-60
cilium config view | grep -E '^bpf-lb-mode |^bpf-lb-algorithm '
k annotate svc echo-np service.cilium.io/lb-algorithm=maglev   # per Service
  • kube-proxy modes: iptables, nftables, ipvs (field mode in its ConfigMap).
  • DSR needs native routing (or Geneve); autoDirectNodeRoutes only in the same L2 network, otherwise BGP.

CoreDNS (Days 4, 8)

k -n kube-system get cm coredns -o yaml > coredns-vorher.yaml      # back up
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
vim Corefile
coredns -conf Corefile 2>&1 | tail -1         # check syntax locally (binary Day 8)
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -
k -n kube-system edit cm coredns              # alternative in the exam
k -n kube-system logs -l k8s-app=kube-dns -f --tail=0 --prefix   # Reloading complete
k -n kube-system rollout restart deploy coredns                  # if in a hurry
sudo grep -A1 -E 'clusterDNS|clusterDomain|resolvConf' /var/lib/kubelet/config.yaml

Blocks in the Corefile – log and hosts/rewrite go into the .:53 block (below ready), the stub domain as its own server block at the end:

    log
    hosts {
       172.16.0.123 router.lab.local
       fallthrough
    }
    rewrite name exact notes.lab.local web.tag08.svc.cluster.local

lab.intern:53 {
    errors
    cache 30
    forward . 172.16.0.123
}
  • Without fallthrough, hosts answers all other names with NXDOMAIN.
  • A missing } makes CoreDNS keep the old configuration ([ERROR] in the log).

Testing names – dig only uses the search list with +search:

k exec client -- dig +short web.tag08.svc.cluster.local          # Service
k exec client -- dig +short web-hl.tag08.svc.cluster.local       # headless: Pod IPs
k exec client -- dig +short +search web
k exec client -- dig +short SRV _http._tcp.web.tag08.svc.cluster.local
k exec client -- dig +short @<coredns-pod-ip> test.lab.local     # ask the Pod directly

dnsPolicy and dnsConfig in the Pod:

spec:
  dnsConfig:
    options:
    - name: ndots
      value: "1"
---
spec:
  dnsPolicy: None                             # only what is in dnsConfig
  dnsConfig:
    nameservers: ["1.1.1.1"]
    searches: ["example.org"]
  • Pod name in DNS (db.web-hl.tag08.svc…): hostname + subdomain = headless Service (Day 8).

EndpointSlices and availability (Day 9)

JP='{range .items[*].endpoints[*]}{.targetRef.name}{" r="}{.conditions.ready}'\
'{" s="}{.conditions.serving}{" t="}{.conditions.terminating}{"\n"}{end}'
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{.items[*].endpoints[?(@.conditions.ready==false)].targetRef.name}'; echo
k expose deployment web --name=web-peers --cluster-ip=None --port=8080 \
  $do > web-peers.yaml
sed -i '/^  clusterIP: None/a\  publishNotReadyAddresses: true' web-peers.yaml
k label node worker1 topology.kubernetes.io/zone=zone-a
k patch svc web -p '{"spec":{"trafficDistribution":"PreferSameZone"}}'
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{range .items[*].endpoints[*]}{.nodeName} {.zone} {.hints}{"\n"}{end}'
k create pdb web --selector=app=web --min-available=1
k get pdb web -o jsonpath='{.status.disruptionsAllowed}'; echo
  • The ENDPOINTS column hides unready Pods – read the conditions. Cilium shows them in cilium-dbg service list as maintenance or terminating.
  • trafficDistribution: PreferSameZone, PreferSameNode, PreferClose (old name); zone hints only with the label topology.kubernetes.io/zone on the nodes.

ClusterMesh (Day 17)

k --context ost -n kube-system get secret cilium-ca \
  -o jsonpath='{.data.ca\.crt}'               # shared CA for west
cilium config view --context west | grep -E '^cluster-(name|id) '
cilium clustermesh enable --context ost --service-type NodePort
cilium clustermesh enable --context west --service-type NodePort
cilium clustermesh connect --context ost --destination-context west
cilium clustermesh status --context ost --wait
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep ClusterMesh

Annotations on the Service (same name and namespace in both clusters):

k annotate svc echo service.cilium.io/global="true"
k --context west annotate svc echo service.cilium.io/global="true"
k annotate svc echo service.cilium.io/affinity=local   # local first, else remote
k --context west annotate svc echo service.cilium.io/shared="false"   # do not share
k annotate svc echo service.cilium.io/affinity-                      # remove
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  --clustermesh-affinity | grep -A1 $CIP

Policy across cluster boundaries – without the cluster label, fromEndpoints only applies locally:

  ingress:
  - fromEndpoints:
    - matchLabels:
        run: client
        io.cilium.k8s.policy.cluster: ost

MCS API (clustermesh.mcsapi.enabled=true, in west additionally clustermesh.mcsapi.corednsAutoConfigure.enabled=true); the Service is exported by a ServiceExport (multicluster.x-k8s.io/v1beta1) with the name and namespace of the Service (Day 17):

k --context west get serviceexport,serviceimport,svc
k --context west exec client -- curl -s echo.tag17.svc.clusterset.local/hostname

Advanced Traffic Management (Days 10, 11, 15, 16, 18)

Switching on Gateway API (Day 10):

GWAPI=https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.3
k apply --server-side -f $GWAPI/standard-install.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
  --reuse-values --set gatewayAPI.enabled=true
k -n kube-system rollout restart deployment/cilium-operator
k -n kube-system rollout restart ds/cilium
k get gatewayclass
k explain httproute.spec.rules                # no k create for Gateway objects

Gateway as NodePort instead of LoadBalancer (without LB IPAM it otherwise never becomes Programmed): a CiliumGatewayClassConfig (cilium.io/v2alpha1) with spec.service.type: NodePort and a GatewayClass of its own – complete in Day 10:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: tag10-nodeport
spec:
  controllerName: io.cilium/gateway-controller
  parametersRef:
    group: cilium.io
    kind: CiliumGatewayClassConfig
    name: nodeport
    namespace: tag10

HTTPRoute with header, path and weight:

spec:                                         # HTTPRoute match (Day 10)
  parentRefs:
  - name: web
  hostnames:
  - shop.example.com
  rules:
  - matches:
    - headers:
      - name: x-version
        value: v2
    backendRefs:
    - name: echo-v2
      port: 8080
  - matches:
    - path:
        type: PathPrefix                      # or Exact
        value: /api/v1
    backendRefs:
    - name: echo-v1
      port: 8080
  - backendRefs:                              # canary 90/10
    - name: echo-v1
      port: 8080
      weight: 90
    - name: echo-v2
      port: 8080
      weight: 10

Filters (each under rules[].filters; URLRewrite and RequestMirror: Day 10):

- type: RequestRedirect
  requestRedirect:
    path:
      type: ReplacePrefixMatch
      replacePrefixMatch: /new
    statusCode: 301
- type: RequestHeaderModifier
  requestHeaderModifier:
    add:
    - name: x-team
      value: blue                             # alongside: set: and remove:

ReferenceGrant in the namespace of the target (route in tag10, Service in tag10-backend):

spec:                                         # ReferenceGrant in tag10-backend
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: tag10
  to:
  - group: ""
    kind: Service

Reading the status and testing:

NP=$(k get svc cilium-gateway-web -o jsonpath='{.spec.ports[0].nodePort}')
G=172.16.0.228:$NP
curl -s -H 'Host: shop.example.com' http://$G/v2/cart
k get gateway web -o jsonpath=\
'{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
RC='{range .status.parents[*].conditions[*]}{.type}={.status} '\
'{.reason}: {.message}{"\n"}{end}'
k get httproute billing -o jsonpath="$RC"     # Accepted, ResolvedRefs
k get gateway web -o jsonpath='{.status.listeners[0].attachedRoutes}{"\n"}'
  • GRPCRoute: Service port with appProtocol: kubernetes.io/h2c, match via matches[].method.service.

TLS at the Gateway (Day 11)

The Certificate (cert-manager) writes the Secret shop-tls, the HTTPS listener refers to it:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: web
spec:
  gatewayClassName: tag11-nodeport
  listeners:
  - name: http
    protocol: HTTP
    port: 80
  - name: https-shop
    protocol: HTTPS
    port: 443
    hostname: shop.example.com
    tls:
      mode: Terminate
      certificateRefs:
      - name: shop-tls                        # other namespace: namespace: …
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: to-https
spec:
  parentRefs:
  - name: web
    sectionName: http                         # attach the route to one listener
  rules:
  - filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        statusCode: 301

Secret in a foreign namespace: ReferenceGrant there, from = Gateway (group gateway.networking.k8s.io, namespace tag11), to = Secret (group "").

k wait --for=condition=Ready certificate shop
k get secret shop-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
NPS=$(k get svc cilium-gateway-web -o jsonpath='{.spec.ports[?(@.port==443)].nodePort}')
R="--resolve shop.example.com:$NPS:172.16.0.228"
curl -s $R --cacert ca.crt https://shop.example.com:$NPS/
k get secret shop-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -ext subjectAltName   # names in the certificate
cmctl renew shop
cmctl status certificate shop

BackendTLSPolicy – TLS from the Gateway all the way to the Pod:

apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: secure
spec:
  targetRefs:
  - group: ""
    kind: Service
    name: secure
  validation:
    caCertificateRefs:
    - group: ""
      kind: ConfigMap                         # k create configmap lab-ca --from-file
      name: lab-ca
    hostname: secure.tag11.svc
  • The annotation cert-manager.io/cluster-issuer: tag11-ca on the Gateway makes cert-manager create the certificate for the listener by itself.

LB IPAM and BGP (Day 15)

Four objects, one chain: ClusterConfig → PeerConfig by name, PeerConfig → Advertisement by label, Advertisement → Service by label; the pool selects by namespace. Pool (CiliumLoadBalancerIPPool) and ClusterConfig with spec only:

spec:                                         # CiliumLoadBalancerIPPool
  blocks:
  - cidr: 10.50.0.0/24
  serviceSelector:
    matchLabels:
      io.kubernetes.service.namespace: tag15
---
spec:                                         # CiliumBGPClusterConfig
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
  bgpInstances:
  - name: ost
    localASN: 65010
    peers:
    - name: router
      peerASN: 65000
      peerAddress: 172.16.0.123
      peerConfigRef:
        name: tag15-peer
---
spec:                                         # CiliumBGPPeerConfig tag15-peer
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: tag15                      # must match the Advertisement
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: tag15-lb
  labels:
    advertise: tag15
spec:
  advertisements:
  - advertisementType: Service
    service:
      addresses:
      - LoadBalancerIP                        # or ClusterIP
    selector:
      matchLabels:
        bgp: "on"
  - advertisementType: PodCIDR
k get ippools                                 # CiliumLoadBalancerIPPools
k label svc echo-lb bgp=on
k annotate svc echo-fix lbipam.cilium.io/ips=10.50.0.50   # fixed LB IP
cilium bgp peers                              # established, Advertised > 0
cilium bgp routes advertised ipv4 unicast --node worker1
k get ciliumbgpnodeconfigs
k patch ciliumbgppeerconfig tag15-peer --type=merge \
  -p '{"spec":{"timers":{"holdTimeSeconds":9,"keepAliveTimeSeconds":3}}}'
ssh [email protected] "sudo vtysh -c 'show bgp summary'"
ssh [email protected] "sudo vtysh -c 'show bgp ipv4 unicast'"
ssh [email protected] "ip route | grep proto.bgp"
  • Advertised 0 with the session up: the PeerConfig’s label selector matches no CiliumBGPAdvertisement, or its Service selector matches no Service.

Egress Gateway (Day 16)

Prerequisite: egressGateway.enabled=true, bpf.masquerade=true, KPR.

spec:                                         # CiliumEgressGatewayPolicy tag16-egw
  selectors:
  - podSelector:
      matchLabels:
        app: egress-demo
        io.kubernetes.pod.namespace: tag16
  destinationCIDRs:
  - 172.16.0.123/32
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"
k label node worker1 egress-node=true
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf egress list  # gateway node
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep Masquerading                         # BPF instead of iptables
k patch cegp tag16-egw --type=merge \
  -p '{"spec":{"egressGateway":{"egressIP":"172.16.0.228"}}}'
k patch cegp tag16-egw --type=merge -p \
  '{"spec":{"destinationCIDRs":["172.16.0.0/24"],"excludedCIDRs":["172.16.0.123/32"]}}'
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
  --to-ip 172.16.0.123 --last 2 --print-node-name
  • Instead of podSelector, namespaceSelector with kubernetes.io/metadata.name works too.
  • Empty map = selector does not match; Gateway IP Not Found = no node with the label.

LLM traffic (Day 18)

    backendRefs:                              # in the HTTPRoute
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: chat
  • Cilium cannot handle InferencePool and reports InvalidKind on the route; in the book, agentgateway takes over. endpointPickerRef.failureMode: FailOpen or FailClose (Day 18).

Network Security and Policy (Days 12–14)

Default Deny (podSelector: {}, policyTypes: [Ingress, Egress], no rules) with a DNS exception (Day 12):

spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
from:                           from:
- namespaceSelector: {...}      - namespaceSelector: {...}
- podSelector: {...}              podSelector: {...}
  -> OR (two entries)             -> AND (one entry)

ipBlock with an exception and a port range:

  egress:
  - to:
    - ipBlock:
        cidr: 172.16.0.0/24
        except:
        - 172.16.0.123/32
  - to:
    - ipBlock:
        cidr: 172.16.0.123/32
    ports:
    - protocol: TCP
      port: 20
      endPort: 23
k -n tag12 describe networkpolicy api-ingress \
  | sed -n '/Allowing ingress/,/Not affecting/p'
k -n tag12 get cnp                            # short form; cluster-wide: k get ccnp
k -n tag12 get cep                            # CiliumEndpoints with identity
  • Ports in policies are container ports (8080), not Service ports (80).
  • Under Cilium, ipBlock does not match node IPs (identities host, remote-node) – use toEntities for that.

CiliumNetworkPolicy: entities, DNS visibility, FQDN, L7:

spec:                                         # CNP client-fqdn
  endpointSelector:
    matchLabels:
      app: client
  egress:
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s:k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"                   # required for toFQDNs
  - toFQDNs:
    - matchName: example.com
    - matchPattern: "*.kubernetes.io"         # does not match kubernetes.io itself
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
---
spec:                                         # CNP notes-api
  endpointSelector:
    matchLabels:
      app: notes-api
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: client
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/(version|healthz)"          # path is a regex
  • Another HTTP rule: method, path and headers: ["X-API-Key: lab-key-123"].

Deny beats allow (Day 12: next to an ingress rule with fromEntities: cluster); cluster-wide by namespace label:

spec:                                         # CNP notes-api-deny
  endpointSelector:
    matchLabels:
      app: notes-api
  ingressDeny:
  - fromEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: tag12-other
        run: bad
---
spec:                                         # CCNP tag12-no-router-ssh
  endpointSelector:
    matchLabels:
      k8s:io.cilium.k8s.namespace.labels.ckne: tag12
  enableDefaultDeny:                          # otherwise the deny blocks all else
    egress: false
    ingress: false
  egressDeny:
  - toCIDR:
    - 172.16.0.123/32
    toPorts:
    - ports:
      - port: "22"
        protocol: TCP
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg policy get \
  k8s:io.cilium.k8s.policy.name=notes-api k8s:io.cilium.k8s.policy.namespace=tag12
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg fqdn cache list
cilium config view | grep '^enable-policy '
  • L7 no = 403 Access denied, L3/L4 no = timeout (drop in Hubble).

Encryption with WireGuard (Day 13)

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set encryption.enabled=true --set encryption.type=wireguard
k -n kube-system rollout restart ds/cilium
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status | grep Encryption
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg encrypt status
ssh [email protected] "ip -d link show cilium_wg0 | head -3"
ssh [email protected] "sudo timeout 7 tcpdump -nn -i eth0 -A \
  'host 172.16.0.83 and (udp port 8472 or udp port 51871)'"
cilium config view | grep node-encryption-opt-out-labels   # control plane exempt
  • Peers in Encryption = number of other nodes. With WireGuard, UDP 51871 instead of plaintext in VXLAN (UDP 8472).

Identities and mutual auth (Day 14)

k get cep -o custom-columns=NAME:.metadata.name,ID:.status.identity.id
k get ciliumidentity <id> -o yaml | sed -n '/security-labels/,$p'
k label pod frontend team=blue                # new label = new identity
  ingress:
  - authentication:
      mode: required                          # level of fromEndpoints
    fromEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: /notes
S="kubectl -n cilium-spire exec spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server"
$S healthcheck
$S entry show -spiffeID spiffe://spiffe.cilium/identity/<id>
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth list
k explain cnp.spec.ingress.authentication.mode
  • Drop reason Authentication required. Authenticated does not mean encrypted.

Observability (Days 19, 20)

Hubble filters (Days 4, 12, 20) – all hubble commands with -P --port-forward-port 0 (starts the port-forward by itself), omitted here for brevity:

hubble observe -n <ns> --verdict DROPPED --last 5
hubble observe --pod <ns>/<pod> --protocol http --last 4      # both directions
hubble observe --from-pod <ns>/<pod> --to-port 53 --last 2    # one direction only
hubble observe --to-pod <ns>/<pod> --verdict DROPPED --last 2
hubble observe --to-namespace <ns> --to-label version=v2 --type l7 --last 2
hubble observe --to-ip 172.16.0.123 --print-node-name --last 2
hubble observe --http-status 4+ --since 5m
hubble observe --verdict DROPPED --drop-reason-desc POLICY_DENIED --last 5   # Day 21
hubble observe --http-path '/customer' --protocol http --last 2
hubble observe --trace-id <trace-id> --type l7 --last 50
hubble observe --from-cluster ost --to-port 8080 --last 1     # ClusterMesh
hubble observe -f --pod <ns>/<pod> --type l7                  # follow live
hubble observe -P --port-forward-port 0 --kube-context west -n tag17 --last 1
hubble list nodes

Evaluating JSON, filtering a capture offline:

hubble observe --to-pod <ns>/<pod> --verdict DROPPED --last 1 -o json \
  | jq -c '{src: .flow.source.pod_name, dst: .flow.destination.pod_name,
  port: .flow.l4.TCP.destination_port, drop: .flow.drop_reason_desc}'
hubble observe -n tag20 --last 300 -o jsonpb > mitschnitt.json
hubble observe --input-file mitschnitt.json --pod tag20/customer --protocol http
  • No L7 flows without an L7 rule: rules: http: [{}] or dns in a CNP makes the traffic visible without blocking anything.

Flow logs to a file (Helm values, with --reuse-values -f flowlog.yaml):

hubble:
  export:
    dynamic:
      enabled: true
      config:
        content:
        - name: tag20
          filePath: /var/run/cilium/hubble/tag20.log
          includeFilters:
          - source_pod: ["tag20/"]
          - destination_pod: ["tag20/"]
          fileMaxSizeMb: 5
          fileMaxBackups: 2
cilium config view | grep flowlogs
N=$(k get pod <pod> -o jsonpath='{.status.hostIP}')   # node IP of the Pod
ssh ubuntu@$N "sudo cat /var/run/cilium/hubble/tag20.log" > w2.log
jq -r '.flow.verdict' w2.log | sort | uniq -c
jq -r 'select(.flow.verdict=="DROPPED") | .flow.drop_reason_desc' w2.log | uniq -c

Metrics (Day 19)

Helm values (Day 19, via -f): prometheus.enabled and .serviceMonitor.enabled, hubble.metrics.enabled as a list (dns, drop, tcp, flow, port-distribution, httpV2:labelsContext=source_namespace,source_workload,…) and hubble.metrics.serviceMonitor.enabled.

cilium config view | grep -E 'hubble-metrics |prometheus-serve-addr'
curl -s localhost:9962/metrics | grep '^cilium_drop_count_total'   # agent, per node
curl -s localhost:9965/metrics | grep '^hubble_drop_total'         # Hubble, per node
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list \
  -p drop_count
k -n kube-system get servicemonitors

PromQL (in the book via promtool query instant in the Prometheus Pod):

count by (job) (up{job=~"cilium.*|hubble.*"})
sum by (reason, node) (rate(hubble_drop_total[5m]))
sum by (rcode) (rate(hubble_dns_responses_total[1m]))
sum by (status) (rate(hubble_http_requests_total{destination_namespace="tag19"}[1m]))
histogram_quantile(0.95, sum by (le, destination_workload)
  (rate(hubble_http_request_duration_seconds_bucket[5m])))

Alert as a PrometheusRule (spec.groups[].rules[] with alert, expr, for; Day 19):

expr: sum by (node) (rate(hubble_drop_total{reason="POLICY_DENIED"}[1m])) > 0.2
  • Drop reasons are terse in hubble_drop_total (POLICY_DENIED) and spelled out in cilium_drop_count_total (Policy denied).
  • DNS and HTTP metrics only exist for traffic with an L7 rule.

Symptom and first command

Symptom First command
Node NotReady, “cni plugin not initialized” /etc/cni/net.d, agent Pod
Who implements Services? cilium-dbg status, line KubeProxyReplacement
Service does not answer k get endpointslices -l kubernetes.io/service-name=…
Endpoints empty selector against --show-labels, targetPort against container
Pod gets RST cilium-dbg service list: backend port
Name does not resolve cat /etc/resolv.conf in the Pod, dig with the full name
Corefile change has no effect CoreDNS logs: Reloading complete or [ERROR]
Connection hangs (timeout) hubble observe --verdict DROPPED
403 Access denied L7 rule of the CNP (method, path, header)
toFQDNs blocks everything DNS rule with dns: matchPattern is missing
LoadBalancer <pending> k get ippools, serviceSelector
Gateway not Programmed Gateway conditions, Service type (NodePort class)
Route not Accepted route conditions: sectionName, allowedRoutes
ResolvedRefs=False Service name, ReferenceGrant in the target namespace
TLS listener fails .status.listeners[*].conditions: Secret name
BGP route missing on the router cilium bgp peers: Advertised, labels
Egress Gateway has no effect cilium-dbg bpf egress list on the gateway node
WireGuard set, Encryption: Disabled rollout restart ds/cilium
Clusters do not see each other cilium clustermesh status, cluster-name/-id
No Hubble metric cilium config view: hubble-metrics, port 9965
No L7 flows in Hubble CNP with http: [{}] or dns

Docs pages

The pages listed in the days’ Exam Tips, by domain:

  • Craft: kubernetes.io “Debug Running Pods”; docs.cilium.io “Troubleshooting”, “Command Cheatsheet”
  • CNI, IPAM: kubernetes.io “Network Plugins”; docs.cilium.io “Installation using Helm”, “Upgrade Guide”, “IP Address Management (IPAM)”
  • Multus: github.com/k8snetworkplumbingwg/multus-cni (“Quickstart”, “How to use”)
  • Services: kubernetes.io “Service”, “Service Internal Traffic Policy”, “Create an External Load Balancer”, “Virtual IPs and Service Proxies”
  • kube-proxy replacement: docs.cilium.io “Kubernetes Without kube-proxy”, “Routing”
  • DNS: kubernetes.io “Customizing DNS Service”, “DNS for Services and Pods”, “Debugging DNS Resolution”; coredns.io/plugins
  • Endpoints: kubernetes.io “EndpointSlices”, “Traffic Distribution”, “Specifying a Disruption Budget for your Application”
  • Gateway API: gateway-api.sigs.k8s.io “Guides”, “TLS”, “BackendTLSPolicy”; cert-manager.io “Gateway”; docs.cilium.io “Gateway API – HTTPS Example”
  • BGP, LB IPAM: docs.cilium.io “BGP Control Plane”, “LoadBalancer IP Address Management (LB IPAM)”, “L2 Announcements”
  • Egress: docs.cilium.io “Egress Gateway”
  • ClusterMesh: docs.cilium.io “Setting up Cluster Mesh”, “Load-balancing & Service Discovery”
  • LLM traffic: gateway-api-inference-extension.sigs.k8s.io “API Overview”, “Implementations”
  • Policies: kubernetes.io “Network Policies”; docs.cilium.io “Layer 3/4/7 Examples”, “DNS based”, “Deny Policies”
  • Encryption, auth: docs.cilium.io “WireGuard Transparent Encryption”, “Mutual Authentication (Beta)”
  • Observability: docs.cilium.io “Monitoring & Metrics”, “Inspecting Network Flows with the CLI”, “Configuring Hubble exporter”, “Layer 7 Protocol Visibility”

Which of these pages you may open during the exam is stated in the Linux Foundation’s current exam instructions.