Appendix A: Cheat Sheet
The most important commands and snippets from all days, ordered by
curriculum domain. The days are in parentheses. You replace the
placeholders in angle brackets; IPs are those of the lab
(cp1 172.16.0.126, worker1 .228,
worker2 .83, router .123, west
.182).
Exam Craft (Days 0, 1)
source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k
export do="--dry-run=client -o yaml" # generate YAML instead of creating
export now="--force --grace-period=0" # delete immediately
source <(cilium completion bash)
source <(hubble completion bash)
k config use-context <context> # first action of every task
k config set-context --current --namespace=<ns>
k config view --minify -o jsonpath='{..namespace}'; echo
ssh [email protected] # node work, exit at the endAlways run cilium-dbg in the agent of the affected node
– ds/cilium hits a random one:
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=<node> -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --brief
N=$(k get pod <pod> -o jsonpath='{.spec.nodeName}') # node of a Pod
hubble status -P --port-forward-port 0 # -P: port-forward to Relay per callLinux tools (Days 1, 4)
ip -br link
ip -br addr show cilium_host
ip route
ip route get 10.0.2.248 # which route does this IP take?
sudo ss -tlnp '( sport = :6443 or sport = :10250 or sport = :4240 or sport = :4244 )'
sudo iptables-save | grep -c KUBE-SVC # 0 = no kube-proxy
sudo nft list tables
sudo tcpdump -i any -nn -c 4 host 10.0.2.248 and tcp port 80
sudo tcpdump -l -i eth0 -nn udp port 8472 # VXLAN between nodes
sudo tcpdump -nn -e -i eth0 -c 2 tcp port 30707 # -e shows MAC addresses (DSR)Into a Pod’s network (on the Pod’s node):
sudo crictl pods --namespace <ns> --name <pod>
ID=$(sudo crictl ps --name <container> -q --label io.kubernetes.pod.namespace=<ns>)
P=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
sudo nsenter -t $P -n ip -br addr
sudo nsenter -t $P -n ip route
sudo nsenter -t $P -n ss -tlnp
sudo nsenter -t $P -n tcpdump -i eth0 -nn -c 2 icmp
ip -o link | awk -F': ' '$1==33{print $2}' # host end for eth0@if33kubectl debug with netshoot, without SSH:
k debug <pod> -it --image=nicolaka/netshoot:v0.16 --target=<container> \
--profile=netadmin # ephemeral container in the Pod netns
k debug node/worker2 -it --image=nicolaka/netshoot:v0.16 --profile=netadmin
# afterwards delete the Pod node-debugger-worker2-…, it stays behind
k run c1 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinityCore Infrastructure and CNI (Days 2–5)
CNI on the node (Day 2):
sudo ls -la /etc/cni/net.d # drwx------, only with sudo
sudo cat /etc/cni/net.d/05-cilium.conflist
ls /opt/cni/bin
sudo crictl info | jq '.cniconfig | {PluginConfDir, PluginDirs, PluginMaxConfNum}'
sudo journalctl -u containerd --since -1min | grep -m1 'reload cni'
cilium config view | grep -E '^cni-|write-cni'
k -n kube-system get cm cilium-config -o jsonpath='{.data.cni-exclusive}'; echo
P=$(sudo crictl pods --name <pod> --namespace <ns> -q)
sudo crictl inspectp $P | jq -c '.info.cniResult.Interfaces.eth0.IPConfigs'- If
05-cilium.conflistis missing, the agent rewrites it (Wrote CNIin the log). - With
cni.exclusive=true, Cilium renames foreign files to*.cilium_bak.
Changing Helm values (pattern for every day)
helm get values cilium -n kube-system -o yaml > values-vorher.yaml # back up first
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
--reuse-values --set <key>=<value> # without --reuse-values: all reset
k -n kube-system rollout restart ds/cilium # for operator values also
k -n kube-system rollout restart deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep <key> # result in the ConfigMap
helm get values cilium -n kube-system --all | grep -A13 '^cni:' # incl. defaults
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
-f values-vorher.yaml # way back
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICHcilium config set debug true --restart=falseonly changes the ConfigMap – drift that Helm 4 reports as a conflict on rollback (helm rollback … --force-conflicts).- Values from a file:
helm upgrade … --reuse-values -f datei.yaml.
| Helm value | Effect | Day |
|---|---|---|
debug.enabled=true |
debug logs in the agent | 2 |
ipam.operator.clusterPoolIPv4PodCIDRList |
Pod pool, only extend | 3 |
cni.exclusive=false |
tolerate foreign CNI files (Multus) | 5 |
routingMode=native,
autoDirectNodeRoutes=true |
native routing | 7 |
ipv4NativeRoutingCIDR=10.0.0.0/16 |
network without masquerading | 7 |
loadBalancer.mode=dsr,
loadBalancer.algorithm=maglev |
DSR, Maglev | 7 |
gatewayAPI.enabled=true |
Gateway controller | 10 |
encryption.enabled=true,
encryption.type=wireguard |
WireGuard | 13 |
encryption.nodeEncryption=true |
node-to-node too | 13 |
authentication.mutual.spire.enabled=true |
mutual auth with SPIRE | 14 |
bgpControlPlane.enabled=true |
BGP Control Plane | 15 |
egressGateway.enabled=true,
bpf.masquerade=true |
Egress Gateway | 16 |
cluster.name=ost, cluster.id=1 |
prerequisite for ClusterMesh | 17 |
clustermesh.mcsapi.enabled=true |
MCS API | 17 |
hubble.export.dynamic.enabled=true |
flow logs to a file | 20 |
Values per node (Day 21, round 1) – CiliumNodeConfig
overrides the ConfigMap:
k get ciliumnodeconfigs -A # spec.nodeSelector + spec.defaults
k -n kube-system logs $CP -c config | grep 'Got configuration' # sources readcilium config viewonly reads the ConfigMap and does not show the override.
IPAM (Day 3)
cilium config view | grep -E '^ipam |cluster-pool'
k get ciliumnodes -o custom-columns='NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs'
k get ciliumnodes -o jsonpath=\
'{range .items[*]}{.metadata.name}{"\t"}{.status.ipam.operator-status}{"\n"}{end}'
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --all-addresses \
| sed -n '/^IPAM/,/^IPv4 BIG/p'
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache get <ip>
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg ip get -v <ip>/32
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set ipam.operator.clusterPoolIPv4PodCIDRList='{10.0.0.0/16,10.2.0.0/16}'
k -n kube-system rollout restart deployment cilium-operatornode.spec.podCIDRis only filled in IPAM modekubernetes– withcluster-pool, the CiliumNode is what counts.CiliumPodIPPoolplus the annotationipam.cilium.io/ip-poolonly takes effect withipam.mode=multi-pool(created in the lab, but without effect).
Debugging Pod connectivity (Day 4)
Order: Pod Ready? → Pod IP directly → EndpointSlices →
DNS in the Pod → drops.
k get pods -o wide
k exec c1 -- curl -s -o /dev/null -w '%{remote_ip} %{http_code}\n' http://<pod-ip>
k exec c1 -- traceroute -n -w 1 -q 1 <pod-ip>
k get endpointslices
k describe svc <svc> | grep -E 'Selector|Endpoints'
k get pods --show-labels
k exec c1 -- cat /etc/resolv.conf
k exec c1 -- nslookup web
hubble observe -P --port-forward-port 0 --pod <ns>/c1 --tcp-flags SYN --last 2
cilium config view | grep -E '^routing-mode|^tunnel-protocol|^auto-direct'
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg monitor --related-to <ep-id>
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg identity get <id>
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg node list
cilium connectivity test --test-namespace tag04-ct --test no-policies
k set selector svc web-sel app=web # repair
k patch svc api -p '{"spec":{"ports":[{"port":80,"targetPort":"web"}]}}' # port name- No flow in Hubble = no backend;
RST= wrong port;0/1 Ready= probe.
Multus (Day 5)
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-lab
spec:
config: |
{"cniVersion": "0.3.1", "type": "macvlan", "master": "eth0", "mode": "bridge",
"capabilities": {"ips": true}, "ipam": {"type": "static"}}k get net-attach-def -A
NET1='[{"name":"macvlan-lab","ips":["192.168.77.11/24"]}]'
k run m1 --image=nicolaka/netshoot:v0.16 \
--annotations=k8s.v1.cni.cncf.io/networks="$NET1" -- sleep infinity
k exec m1 -- ip -br addr # eth0 = Cilium, net1 = Multus
k get pod m1 \
-o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'; echo- JSON form of the annotation:
name,interface,ips; short form only the name.00-multus.confmust be the first file in/etc/cni/net.d.net1bypasses Cilium: no policy, no Hubble.
Service Networking and DNS (Days 6–9, 17)
Services at L4 (Day 6):
k expose deployment echo --port=80 --target-port=8080
k expose deployment echo --name=echo-np --type=NodePort --port=80 --target-port=8080
k patch svc echo-np -p '{"spec":{"ports":[{"port":80,"nodePort":30080}]}}'
k expose deployment echo --name=echo-lb --type=LoadBalancer --port=80 \
--target-port=8080
k expose deployment echo --name=echo-hl --cluster-ip=None --port=8080
k create service externalname docs --external-name=kubernetes.io
k patch svc echo -p '{"spec":{"sessionAffinity":"ClientIP"}}'
k patch svc echo-np -p '{"spec":{"externalTrafficPolicy":"Local"}}'
k patch svc echo -p '{"spec":{"internalTrafficPolicy":"Local"}}'
k get svc echo-lb -o jsonpath='{.spec.healthCheckNodePort}' # with ETP LocalService without a selector, with its own EndpointSlice:
k create service clusterip router --tcp=80:8000 $do > router-svc.yaml
sed -i '/selector:/,+1d' router-svc.yamlapiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: router-1
labels:
kubernetes.io/service-name: router
endpointslice.kubernetes.io/managed-by: tag06.example.com
addressType: IPv4
ports:
- name: 80-8000 # same name as in the Service port
protocol: TCP
port: 8000
endpoints:
- addresses:
- 172.16.0.123externalTrafficPolicyonly affects NodePort/LoadBalancer,internalTrafficPolicyonly traffic from inside the cluster;Localwithout a local backend = no answer.
kube-proxy and Cilium KPR (Days 0, 7)
k -n kube-system get cm kube-proxy # NotFound = no kube-proxy
k get pods -A -l k8s-app=kube-proxy
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep KubeProxyReplacement
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --verbose \
| sed -n '/KubeProxyReplacement Details/,/HostPort/p'
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}')
X="kubectl -n kube-system exec $CP -c cilium-agent --" # kubectl, not the alias
$X cilium-dbg service list | grep -A1 "$CIP:"
$X cilium-dbg bpf lb list | grep "$CIP:80"
$X cilium-dbg map list | grep -E '^Name|lb4_'
$X cilium-dbg bpf lb maglev list | cut -c1-60
cilium config view | grep -E '^bpf-lb-mode |^bpf-lb-algorithm '
k annotate svc echo-np service.cilium.io/lb-algorithm=maglev # per Service- kube-proxy modes:
iptables,nftables,ipvs(fieldmodein its ConfigMap). - DSR needs native routing (or Geneve);
autoDirectNodeRoutesonly in the same L2 network, otherwise BGP.
CoreDNS (Days 4, 8)
k -n kube-system get cm coredns -o yaml > coredns-vorher.yaml # back up
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
vim Corefile
coredns -conf Corefile 2>&1 | tail -1 # check syntax locally (binary Day 8)
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -
k -n kube-system edit cm coredns # alternative in the exam
k -n kube-system logs -l k8s-app=kube-dns -f --tail=0 --prefix # Reloading complete
k -n kube-system rollout restart deploy coredns # if in a hurry
sudo grep -A1 -E 'clusterDNS|clusterDomain|resolvConf' /var/lib/kubelet/config.yamlBlocks in the Corefile – log and
hosts/rewrite go into the .:53
block (below ready), the stub domain as its own server
block at the end:
log
hosts {
172.16.0.123 router.lab.local
fallthrough
}
rewrite name exact notes.lab.local web.tag08.svc.cluster.local
lab.intern:53 {
errors
cache 30
forward . 172.16.0.123
}
- Without
fallthrough,hostsanswers all other names with NXDOMAIN. - A missing
}makes CoreDNS keep the old configuration ([ERROR]in the log).
Testing names – dig only uses the search list with
+search:
k exec client -- dig +short web.tag08.svc.cluster.local # Service
k exec client -- dig +short web-hl.tag08.svc.cluster.local # headless: Pod IPs
k exec client -- dig +short +search web
k exec client -- dig +short SRV _http._tcp.web.tag08.svc.cluster.local
k exec client -- dig +short @<coredns-pod-ip> test.lab.local # ask the Pod directlydnsPolicy and dnsConfig in the Pod:
spec:
dnsConfig:
options:
- name: ndots
value: "1"
---
spec:
dnsPolicy: None # only what is in dnsConfig
dnsConfig:
nameservers: ["1.1.1.1"]
searches: ["example.org"]- Pod name in DNS (
db.web-hl.tag08.svc…):hostname+subdomain= headless Service (Day 8).
EndpointSlices and availability (Day 9)
JP='{range .items[*].endpoints[*]}{.targetRef.name}{" r="}{.conditions.ready}'\
'{" s="}{.conditions.serving}{" t="}{.conditions.terminating}{"\n"}{end}'
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath="$JP"
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{.items[*].endpoints[?(@.conditions.ready==false)].targetRef.name}'; echo
k expose deployment web --name=web-peers --cluster-ip=None --port=8080 \
$do > web-peers.yaml
sed -i '/^ clusterIP: None/a\ publishNotReadyAddresses: true' web-peers.yaml
k label node worker1 topology.kubernetes.io/zone=zone-a
k patch svc web -p '{"spec":{"trafficDistribution":"PreferSameZone"}}'
k get endpointslices -l kubernetes.io/service-name=web -o jsonpath=\
'{range .items[*].endpoints[*]}{.nodeName} {.zone} {.hints}{"\n"}{end}'
k create pdb web --selector=app=web --min-available=1
k get pdb web -o jsonpath='{.status.disruptionsAllowed}'; echo- The
ENDPOINTScolumn hides unready Pods – read the conditions. Cilium shows them incilium-dbg service listasmaintenanceorterminating. trafficDistribution:PreferSameZone,PreferSameNode,PreferClose(old name); zone hints only with the labeltopology.kubernetes.io/zoneon the nodes.
ClusterMesh (Day 17)
k --context ost -n kube-system get secret cilium-ca \
-o jsonpath='{.data.ca\.crt}' # shared CA for west
cilium config view --context west | grep -E '^cluster-(name|id) '
cilium clustermesh enable --context ost --service-type NodePort
cilium clustermesh enable --context west --service-type NodePort
cilium clustermesh connect --context ost --destination-context west
cilium clustermesh status --context ost --wait
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep ClusterMeshAnnotations on the Service (same name and namespace in both clusters):
k annotate svc echo service.cilium.io/global="true"
k --context west annotate svc echo service.cilium.io/global="true"
k annotate svc echo service.cilium.io/affinity=local # local first, else remote
k --context west annotate svc echo service.cilium.io/shared="false" # do not share
k annotate svc echo service.cilium.io/affinity- # remove
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
--clustermesh-affinity | grep -A1 $CIPPolicy across cluster boundaries – without the cluster label,
fromEndpoints only applies locally:
ingress:
- fromEndpoints:
- matchLabels:
run: client
io.cilium.k8s.policy.cluster: ostMCS API (clustermesh.mcsapi.enabled=true, in west
additionally
clustermesh.mcsapi.corednsAutoConfigure.enabled=true); the
Service is exported by a ServiceExport
(multicluster.x-k8s.io/v1beta1) with the name and namespace
of the Service (Day 17):
k --context west get serviceexport,serviceimport,svc
k --context west exec client -- curl -s echo.tag17.svc.clusterset.local/hostnameAdvanced Traffic Management (Days 10, 11, 15, 16, 18)
Switching on Gateway API (Day 10):
GWAPI=https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.3
k apply --server-side -f $GWAPI/standard-install.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
--reuse-values --set gatewayAPI.enabled=true
k -n kube-system rollout restart deployment/cilium-operator
k -n kube-system rollout restart ds/cilium
k get gatewayclass
k explain httproute.spec.rules # no k create for Gateway objectsGateway as NodePort instead of LoadBalancer (without LB IPAM it
otherwise never becomes Programmed): a
CiliumGatewayClassConfig (cilium.io/v2alpha1)
with spec.service.type: NodePort and a GatewayClass of its
own – complete in Day 10:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: tag10-nodeport
spec:
controllerName: io.cilium/gateway-controller
parametersRef:
group: cilium.io
kind: CiliumGatewayClassConfig
name: nodeport
namespace: tag10HTTPRoute with header, path and weight:
spec: # HTTPRoute match (Day 10)
parentRefs:
- name: web
hostnames:
- shop.example.com
rules:
- matches:
- headers:
- name: x-version
value: v2
backendRefs:
- name: echo-v2
port: 8080
- matches:
- path:
type: PathPrefix # or Exact
value: /api/v1
backendRefs:
- name: echo-v1
port: 8080
- backendRefs: # canary 90/10
- name: echo-v1
port: 8080
weight: 90
- name: echo-v2
port: 8080
weight: 10Filters (each under rules[].filters; URLRewrite and
RequestMirror: Day 10):
- type: RequestRedirect
requestRedirect:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /new
statusCode: 301
- type: RequestHeaderModifier
requestHeaderModifier:
add:
- name: x-team
value: blue # alongside: set: and remove:ReferenceGrant in the namespace of the target (route
in tag10, Service in tag10-backend):
spec: # ReferenceGrant in tag10-backend
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: tag10
to:
- group: ""
kind: ServiceReading the status and testing:
NP=$(k get svc cilium-gateway-web -o jsonpath='{.spec.ports[0].nodePort}')
G=172.16.0.228:$NP
curl -s -H 'Host: shop.example.com' http://$G/v2/cart
k get gateway web -o jsonpath=\
'{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
RC='{range .status.parents[*].conditions[*]}{.type}={.status} '\
'{.reason}: {.message}{"\n"}{end}'
k get httproute billing -o jsonpath="$RC" # Accepted, ResolvedRefs
k get gateway web -o jsonpath='{.status.listeners[0].attachedRoutes}{"\n"}'- GRPCRoute: Service port with
appProtocol: kubernetes.io/h2c, match viamatches[].method.service.
TLS at the Gateway (Day 11)
The Certificate (cert-manager) writes the Secret
shop-tls, the HTTPS listener refers to it:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: web
spec:
gatewayClassName: tag11-nodeport
listeners:
- name: http
protocol: HTTP
port: 80
- name: https-shop
protocol: HTTPS
port: 443
hostname: shop.example.com
tls:
mode: Terminate
certificateRefs:
- name: shop-tls # other namespace: namespace: …
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: to-https
spec:
parentRefs:
- name: web
sectionName: http # attach the route to one listener
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301Secret in a foreign namespace: ReferenceGrant there,
from = Gateway (group
gateway.networking.k8s.io, namespace tag11),
to = Secret (group "").
k wait --for=condition=Ready certificate shop
k get secret shop-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
NPS=$(k get svc cilium-gateway-web -o jsonpath='{.spec.ports[?(@.port==443)].nodePort}')
R="--resolve shop.example.com:$NPS:172.16.0.228"
curl -s $R --cacert ca.crt https://shop.example.com:$NPS/
k get secret shop-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -ext subjectAltName # names in the certificate
cmctl renew shop
cmctl status certificate shopBackendTLSPolicy – TLS from the Gateway all the way to the Pod:
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
name: secure
spec:
targetRefs:
- group: ""
kind: Service
name: secure
validation:
caCertificateRefs:
- group: ""
kind: ConfigMap # k create configmap lab-ca --from-file
name: lab-ca
hostname: secure.tag11.svc- The annotation
cert-manager.io/cluster-issuer: tag11-caon the Gateway makes cert-manager create the certificate for the listener by itself.
LB IPAM and BGP (Day 15)
Four objects, one chain: ClusterConfig → PeerConfig by name,
PeerConfig → Advertisement by label, Advertisement → Service by label;
the pool selects by namespace. Pool
(CiliumLoadBalancerIPPool) and ClusterConfig with
spec only:
spec: # CiliumLoadBalancerIPPool
blocks:
- cidr: 10.50.0.0/24
serviceSelector:
matchLabels:
io.kubernetes.service.namespace: tag15
---
spec: # CiliumBGPClusterConfig
nodeSelector:
matchLabels:
kubernetes.io/os: linux
bgpInstances:
- name: ost
localASN: 65010
peers:
- name: router
peerASN: 65000
peerAddress: 172.16.0.123
peerConfigRef:
name: tag15-peer
---
spec: # CiliumBGPPeerConfig tag15-peer
families:
- afi: ipv4
safi: unicast
advertisements:
matchLabels:
advertise: tag15 # must match the Advertisement
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
name: tag15-lb
labels:
advertise: tag15
spec:
advertisements:
- advertisementType: Service
service:
addresses:
- LoadBalancerIP # or ClusterIP
selector:
matchLabels:
bgp: "on"
- advertisementType: PodCIDRk get ippools # CiliumLoadBalancerIPPools
k label svc echo-lb bgp=on
k annotate svc echo-fix lbipam.cilium.io/ips=10.50.0.50 # fixed LB IP
cilium bgp peers # established, Advertised > 0
cilium bgp routes advertised ipv4 unicast --node worker1
k get ciliumbgpnodeconfigs
k patch ciliumbgppeerconfig tag15-peer --type=merge \
-p '{"spec":{"timers":{"holdTimeSeconds":9,"keepAliveTimeSeconds":3}}}'
ssh [email protected] "sudo vtysh -c 'show bgp summary'"
ssh [email protected] "sudo vtysh -c 'show bgp ipv4 unicast'"
ssh [email protected] "ip route | grep proto.bgp"Advertised 0with the session up: the PeerConfig’s label selector matches noCiliumBGPAdvertisement, or its Service selector matches no Service.
Egress Gateway (Day 16)
Prerequisite: egressGateway.enabled=true,
bpf.masquerade=true, KPR.
spec: # CiliumEgressGatewayPolicy tag16-egw
selectors:
- podSelector:
matchLabels:
app: egress-demo
io.kubernetes.pod.namespace: tag16
destinationCIDRs:
- 172.16.0.123/32
egressGateway:
nodeSelector:
matchLabels:
egress-node: "true"k label node worker1 egress-node=true
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf egress list # gateway node
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep Masquerading # BPF instead of iptables
k patch cegp tag16-egw --type=merge \
-p '{"spec":{"egressGateway":{"egressIP":"172.16.0.228"}}}'
k patch cegp tag16-egw --type=merge -p \
'{"spec":{"destinationCIDRs":["172.16.0.0/24"],"excludedCIDRs":["172.16.0.123/32"]}}'
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
--to-ip 172.16.0.123 --last 2 --print-node-name- Instead of
podSelector,namespaceSelectorwithkubernetes.io/metadata.nameworks too. - Empty map = selector does not match;
Gateway IP Not Found= no node with the label.
LLM traffic (Day 18)
backendRefs: # in the HTTPRoute
- group: inference.networking.k8s.io
kind: InferencePool
name: chat- Cilium cannot handle InferencePool and reports
InvalidKindon the route; in the book, agentgateway takes over.endpointPickerRef.failureMode:FailOpenorFailClose(Day 18).
Network Security and Policy (Days 12–14)
Default Deny (podSelector: {},
policyTypes: [Ingress, Egress], no rules) with a DNS
exception (Day 12):
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53from: from:
- namespaceSelector: {...} - namespaceSelector: {...}
- podSelector: {...} podSelector: {...}
-> OR (two entries) -> AND (one entry)
ipBlock with an exception and a port range:
egress:
- to:
- ipBlock:
cidr: 172.16.0.0/24
except:
- 172.16.0.123/32
- to:
- ipBlock:
cidr: 172.16.0.123/32
ports:
- protocol: TCP
port: 20
endPort: 23k -n tag12 describe networkpolicy api-ingress \
| sed -n '/Allowing ingress/,/Not affecting/p'
k -n tag12 get cnp # short form; cluster-wide: k get ccnp
k -n tag12 get cep # CiliumEndpoints with identity- Ports in policies are container ports (8080), not Service ports (80).
- Under Cilium,
ipBlockdoes not match node IPs (identitieshost,remote-node) – usetoEntitiesfor that.
CiliumNetworkPolicy: entities, DNS visibility, FQDN, L7:
spec: # CNP client-fqdn
endpointSelector:
matchLabels:
app: client
egress:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s:k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*" # required for toFQDNs
- toFQDNs:
- matchName: example.com
- matchPattern: "*.kubernetes.io" # does not match kubernetes.io itself
toPorts:
- ports:
- port: "443"
protocol: TCP
---
spec: # CNP notes-api
endpointSelector:
matchLabels:
app: notes-api
ingress:
- fromEndpoints:
- matchLabels:
app: client
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: GET
path: "/(version|healthz)" # path is a regex- Another HTTP rule:
method,pathandheaders: ["X-API-Key: lab-key-123"].
Deny beats allow (Day 12: next to an ingress rule with
fromEntities: cluster); cluster-wide by namespace
label:
spec: # CNP notes-api-deny
endpointSelector:
matchLabels:
app: notes-api
ingressDeny:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: tag12-other
run: bad
---
spec: # CCNP tag12-no-router-ssh
endpointSelector:
matchLabels:
k8s:io.cilium.k8s.namespace.labels.ckne: tag12
enableDefaultDeny: # otherwise the deny blocks all else
egress: false
ingress: false
egressDeny:
- toCIDR:
- 172.16.0.123/32
toPorts:
- ports:
- port: "22"
protocol: TCPk -n kube-system exec $CP -c cilium-agent -- cilium-dbg policy get \
k8s:io.cilium.k8s.policy.name=notes-api k8s:io.cilium.k8s.policy.namespace=tag12
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg fqdn cache list
cilium config view | grep '^enable-policy '- L7 no =
403 Access denied, L3/L4 no = timeout (drop in Hubble).
Encryption with WireGuard (Day 13)
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set encryption.enabled=true --set encryption.type=wireguard
k -n kube-system rollout restart ds/cilium
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status | grep Encryption
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg encrypt status
ssh [email protected] "ip -d link show cilium_wg0 | head -3"
ssh [email protected] "sudo timeout 7 tcpdump -nn -i eth0 -A \
'host 172.16.0.83 and (udp port 8472 or udp port 51871)'"
cilium config view | grep node-encryption-opt-out-labels # control plane exempt- Peers in
Encryption= number of other nodes. With WireGuard, UDP 51871 instead of plaintext in VXLAN (UDP 8472).
Identities and mutual auth (Day 14)
k get cep -o custom-columns=NAME:.metadata.name,ID:.status.identity.id
k get ciliumidentity <id> -o yaml | sed -n '/security-labels/,$p'
k label pod frontend team=blue # new label = new identity ingress:
- authentication:
mode: required # level of fromEndpoints
fromEndpoints:
- matchLabels:
io.cilium.k8s.policy.serviceaccount: frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: GET
path: /notesS="kubectl -n cilium-spire exec spire-server-0 -c spire-server -- \
/opt/spire/bin/spire-server"
$S healthcheck
$S entry show -spiffeID spiffe://spiffe.cilium/identity/<id>
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf auth list
k explain cnp.spec.ingress.authentication.mode- Drop reason
Authentication required. Authenticated does not mean encrypted.
Observability (Days 19, 20)
Hubble filters (Days 4, 12, 20) – all hubble commands
with -P --port-forward-port 0 (starts the port-forward by
itself), omitted here for brevity:
hubble observe -n <ns> --verdict DROPPED --last 5
hubble observe --pod <ns>/<pod> --protocol http --last 4 # both directions
hubble observe --from-pod <ns>/<pod> --to-port 53 --last 2 # one direction only
hubble observe --to-pod <ns>/<pod> --verdict DROPPED --last 2
hubble observe --to-namespace <ns> --to-label version=v2 --type l7 --last 2
hubble observe --to-ip 172.16.0.123 --print-node-name --last 2
hubble observe --http-status 4+ --since 5m
hubble observe --verdict DROPPED --drop-reason-desc POLICY_DENIED --last 5 # Day 21
hubble observe --http-path '/customer' --protocol http --last 2
hubble observe --trace-id <trace-id> --type l7 --last 50
hubble observe --from-cluster ost --to-port 8080 --last 1 # ClusterMesh
hubble observe -f --pod <ns>/<pod> --type l7 # follow live
hubble observe -P --port-forward-port 0 --kube-context west -n tag17 --last 1
hubble list nodesEvaluating JSON, filtering a capture offline:
hubble observe --to-pod <ns>/<pod> --verdict DROPPED --last 1 -o json \
| jq -c '{src: .flow.source.pod_name, dst: .flow.destination.pod_name,
port: .flow.l4.TCP.destination_port, drop: .flow.drop_reason_desc}'
hubble observe -n tag20 --last 300 -o jsonpb > mitschnitt.json
hubble observe --input-file mitschnitt.json --pod tag20/customer --protocol http- No L7 flows without an L7 rule:
rules: http: [{}]ordnsin a CNP makes the traffic visible without blocking anything.
Flow logs to a file (Helm values, with
--reuse-values -f flowlog.yaml):
hubble:
export:
dynamic:
enabled: true
config:
content:
- name: tag20
filePath: /var/run/cilium/hubble/tag20.log
includeFilters:
- source_pod: ["tag20/"]
- destination_pod: ["tag20/"]
fileMaxSizeMb: 5
fileMaxBackups: 2cilium config view | grep flowlogs
N=$(k get pod <pod> -o jsonpath='{.status.hostIP}') # node IP of the Pod
ssh ubuntu@$N "sudo cat /var/run/cilium/hubble/tag20.log" > w2.log
jq -r '.flow.verdict' w2.log | sort | uniq -c
jq -r 'select(.flow.verdict=="DROPPED") | .flow.drop_reason_desc' w2.log | uniq -cMetrics (Day 19)
Helm values (Day 19, via -f):
prometheus.enabled and
.serviceMonitor.enabled,
hubble.metrics.enabled as a list (dns,
drop, tcp, flow,
port-distribution,
httpV2:labelsContext=source_namespace,source_workload,…)
and hubble.metrics.serviceMonitor.enabled.
cilium config view | grep -E 'hubble-metrics |prometheus-serve-addr'
curl -s localhost:9962/metrics | grep '^cilium_drop_count_total' # agent, per node
curl -s localhost:9965/metrics | grep '^hubble_drop_total' # Hubble, per node
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list \
-p drop_count
k -n kube-system get servicemonitorsPromQL (in the book via promtool query instant in the
Prometheus Pod):
count by (job) (up{job=~"cilium.*|hubble.*"})
sum by (reason, node) (rate(hubble_drop_total[5m]))
sum by (rcode) (rate(hubble_dns_responses_total[1m]))
sum by (status) (rate(hubble_http_requests_total{destination_namespace="tag19"}[1m]))
histogram_quantile(0.95, sum by (le, destination_workload)
(rate(hubble_http_request_duration_seconds_bucket[5m])))
Alert as a PrometheusRule (spec.groups[].rules[] with
alert, expr, for; Day 19):
expr: sum by (node) (rate(hubble_drop_total{reason="POLICY_DENIED"}[1m])) > 0.2
- Drop reasons are terse in
hubble_drop_total(POLICY_DENIED) and spelled out incilium_drop_count_total(Policy denied). - DNS and HTTP metrics only exist for traffic with an L7 rule.
Symptom and first command
| Symptom | First command |
|---|---|
Node NotReady, “cni plugin not initialized” |
/etc/cni/net.d, agent Pod |
| Who implements Services? | cilium-dbg status, line
KubeProxyReplacement |
| Service does not answer | k get endpointslices -l kubernetes.io/service-name=… |
| Endpoints empty | selector against --show-labels, targetPort
against container |
Pod gets RST |
cilium-dbg service list: backend port |
| Name does not resolve | cat /etc/resolv.conf in the Pod, dig with
the full name |
| Corefile change has no effect | CoreDNS logs: Reloading complete or
[ERROR] |
| Connection hangs (timeout) | hubble observe --verdict DROPPED |
403 Access denied |
L7 rule of the CNP (method, path, header) |
toFQDNs blocks everything |
DNS rule with dns: matchPattern is missing |
LoadBalancer <pending> |
k get ippools, serviceSelector |
Gateway not Programmed |
Gateway conditions, Service type (NodePort class) |
Route not Accepted |
route conditions: sectionName,
allowedRoutes |
ResolvedRefs=False |
Service name, ReferenceGrant in the target namespace |
| TLS listener fails | .status.listeners[*].conditions: Secret name |
| BGP route missing on the router | cilium bgp peers: Advertised, labels |
| Egress Gateway has no effect | cilium-dbg bpf egress list on the gateway node |
WireGuard set, Encryption: Disabled |
rollout restart ds/cilium |
| Clusters do not see each other | cilium clustermesh status,
cluster-name/-id |
| No Hubble metric | cilium config view: hubble-metrics, port
9965 |
| No L7 flows in Hubble | CNP with http: [{}] or dns |
Docs pages
The pages listed in the days’ Exam Tips, by domain:
- Craft: kubernetes.io “Debug Running Pods”; docs.cilium.io “Troubleshooting”, “Command Cheatsheet”
- CNI, IPAM: kubernetes.io “Network Plugins”; docs.cilium.io “Installation using Helm”, “Upgrade Guide”, “IP Address Management (IPAM)”
- Multus: github.com/k8snetworkplumbingwg/multus-cni (“Quickstart”, “How to use”)
- Services: kubernetes.io “Service”, “Service Internal Traffic Policy”, “Create an External Load Balancer”, “Virtual IPs and Service Proxies”
- kube-proxy replacement: docs.cilium.io “Kubernetes Without kube-proxy”, “Routing”
- DNS: kubernetes.io “Customizing DNS Service”, “DNS for Services and Pods”, “Debugging DNS Resolution”; coredns.io/plugins
- Endpoints: kubernetes.io “EndpointSlices”, “Traffic Distribution”, “Specifying a Disruption Budget for your Application”
- Gateway API: gateway-api.sigs.k8s.io “Guides”, “TLS”, “BackendTLSPolicy”; cert-manager.io “Gateway”; docs.cilium.io “Gateway API – HTTPS Example”
- BGP, LB IPAM: docs.cilium.io “BGP Control Plane”, “LoadBalancer IP Address Management (LB IPAM)”, “L2 Announcements”
- Egress: docs.cilium.io “Egress Gateway”
- ClusterMesh: docs.cilium.io “Setting up Cluster Mesh”, “Load-balancing & Service Discovery”
- LLM traffic: gateway-api-inference-extension.sigs.k8s.io “API Overview”, “Implementations”
- Policies: kubernetes.io “Network Policies”; docs.cilium.io “Layer 3/4/7 Examples”, “DNS based”, “Deny Policies”
- Encryption, auth: docs.cilium.io “WireGuard Transparent Encryption”, “Mutual Authentication (Beta)”
- Observability: docs.cilium.io “Monitoring & Metrics”, “Inspecting Network Flows with the CLI”, “Configuring Hubble exporter”, “Layer 7 Protocol Visibility”
Which of these pages you may open during the exam is stated in the Linux Foundation’s current exam instructions.