Day 0: Training Ground
Goal: A kubeadm cluster v1.37.2 with three nodes runs without kube-proxy; Cilium takes over the Pod network, Services and Hubble. · Duration: approx. 90 min · Prerequisite: three VMs with Ubuntu 24.04, a fourth one for the role
router
Briefing
The CKNE tests Kubernetes networking on ready-made clusters. Here you build your own, deliberately without kube-proxy: Cilium replaces it with eBPF (kube-proxy replacement, KPR for short). Cilium assigns Pod IPs itself from 10.0.0.0/16, and packets between nodes travel through a VXLAN tunnel. Curriculum reference: foundation for “CKNE: Core Infrastructure and CNI”.
cp1 .126 worker1 .228 worker2 .83
control-plane, kubectl, worker worker
helm, cilium, hubble
4 vCPU / 11 GB 2 vCPU / 3 GB 2 vCPU / 3 GB
|________________________|_______________________|______ router .123
172.16.0.0/24 · Pods 10.0.0.0/16 (Cilium, VXLAN) (prod, external
Services 10.96.0.0/12 · no kube-proxy client/server from
Day 0, BGP from Day 15)
Replace the IPs with your own. There is no name resolution between
the VMs, which is why IPs appear everywhere. Drills 1 to 3 run on
cp1, worker1 and worker2, the rest on
cp1.
Drill 1: Prepare the kernel
All downloads in this book query the architecture
(dpkg --print-architecture). On a Mac with Apple Silicon
(for example VMs via multipass) that is arm64 – the
commands stay the same, only the outputs in the book show
amd64.
printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<'EOF' | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sudo sysctl --system >/dev/null
sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables
swapon --show | wc -lExpected:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
0
If swapon shows anything: run
sudo swapoff -a and comment out the swap line in
/etc/fstab. Why ip_forward? Without forwarding
in the kernel, no packet gets from a Pod beyond the node.
Save your progress? Sign up for free
Drill 2: containerd and runc
Both straight from the project, each with a checksum:
cd /tmp
V=2.4.2
U=https://github.com/containerd/containerd/releases/download/v$V
ARCH=$(dpkg --print-architecture)
curl -sSLO $U/containerd-$V-linux-${ARCH}.tar.gz
curl -sSLO $U/containerd-$V-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c containerd-$V-linux-${ARCH}.tar.gz.sha256sum
sudo tar Cxzf /usr/local containerd-$V-linux-${ARCH}.tar.gz
sudo mkdir -p /usr/local/lib/systemd/system
sudo curl -sSL -o /usr/local/lib/systemd/system/containerd.service \
https://raw.githubusercontent.com/containerd/containerd/v$V/containerd.service
R=1.5.3
U=https://github.com/opencontainers/runc/releases/download/v$R
curl -sSLO $U/runc.${ARCH}
curl -sSLO $U/runc.sha256sum
grep " runc.${ARCH}$" runc.sha256sum | sha256sum -c
sudo install -m 755 runc.${ARCH} /usr/local/sbin/runc
runc --version | head -1Expected:
containerd-2.4.2-linux-amd64.tar.gz: OK
runc.amd64: OK
runc version 1.5.3
Generate the configuration and switch to the systemd cgroup driver (like the kubelet):
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
grep -n SystemdCgroup /etc/containerd/config.toml
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
containerd --versionExpected:
89: SystemdCgroup = true
containerd github.com/containerd/containerd/v2 v2.4.2 b7a25261e029...
Save your progress? Sign up for free
Drill 3: kubeadm, kubelet, kubectl in v1.37.2
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg jq
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
| sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
K=/etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=$K] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /" \
| sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
apt-cache madison kubeadm | head -2
sudo apt-get install -y kubelet=1.37.2-1.1 kubeadm=1.37.2-1.1 kubectl=1.37.2-1.1 \
cri-tools
sudo apt-mark hold kubelet kubeadm kubectl
printf 'runtime-endpoint: unix:///run/containerd/containerd.sock\n' \
| sudo tee /etc/crictl.yaml
printf 'image-endpoint: unix:///run/containerd/containerd.sock\n' \
| sudo tee -a /etc/crictl.yaml
kubeadm version -o short
sudo crictl versionExpected (abridged):
kubeadm | 1.37.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb Packages
kubeadm | 1.37.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb Packages
...
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
...
v1.37.2
RuntimeName: containerd
RuntimeVersion: v2.4.2
hold prevents an apt upgrade from changing
the cluster version on the side. The package kubernetes-cni
comes along as a dependency – it puts standard plugins into
/opt/cni/bin that Cilium does not need (more on that on Day
2).
Save your progress? Sign up for free
Drill 4: Control plane without kube-proxy
On cp1. --skip-phases=addon/kube-proxy
leaves out the kube-proxy DaemonSet. A --pod-network-cidr
is deliberately missing: with IPAM cluster-pool, Cilium
hands out the Pod networks itself, and the kube-controller-manager
should not write a second, diverging allocation into
node.spec.podCIDR.
sudo kubeadm init --kubernetes-version=v1.37.2 \
--apiserver-advertise-address=172.16.0.126 --skip-phases=addon/kube-proxy
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
mkdir -p ~/ckne/tag00
kubectl get nodes
kubectl -n kube-system get ds
kubectl -n kube-system get cm kubeadm-config \
-o jsonpath='{.data.ClusterConfiguration}' | grep -A1 '^proxy'Expected (abridged):
Your Kubernetes control-plane has initialized successfully!
...
NAME STATUS ROLES AGE VERSION
cp1 NotReady control-plane 64s v1.37.2
No resources found in kube-system namespace.
proxy:
disabled: true
NotReady is correct: without a CNI, no Pod gets an IP.
No DaemonSet means: no kube-proxy. kubeadm remembers
proxy.disabled: true so that kubeadm upgrade
does not install it later either.
The basic exam setup for the shell on cp1 (completion
for kubectl, cilium and hubble works from Drill 5 on):
cat <<'EOF' >> ~/.bashrc
source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k
export do="--dry-run=client -o yaml"
export now="--force --grace-period=0"
source <(cilium completion bash)
source <(hubble completion bash)
EOF
cat <<'EOF' >> ~/.vimrc
set expandtab
set tabstop=2
set shiftwidth=2
EOF$do generates YAML without creating anything;
$now deletes without waiting. The .vimrc
ensures spaces instead of tabs in YAML. Day 1 goes deeper into this. Run
source ~/.bashrc only after Drill 5, otherwise
cilium and hubble are still missing.
Save your progress? Sign up for free
Drill 5: Helm, cilium CLI and hubble CLI
Three tools on cp1, all with a checksum.
--no-same-owner makes sure the files are owned by root and
not by the user ID from the archive:
cd /tmp
ARCH=$(dpkg --print-architecture)
H=v4.3.2
curl -sSLO https://get.helm.sh/helm-$H-linux-${ARCH}.tar.gz
curl -sSLO https://get.helm.sh/helm-$H-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c helm-$H-linux-${ARCH}.tar.gz.sha256sum
tar xzf helm-$H-linux-${ARCH}.tar.gz
sudo install -m 755 linux-${ARCH}/helm /usr/local/bin/helm
C=v0.20.3
U=https://github.com/cilium/cilium-cli/releases/download/$C
curl -sSLO $U/cilium-linux-${ARCH}.tar.gz
curl -sSLO $U/cilium-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c cilium-linux-${ARCH}.tar.gz.sha256sum
sudo tar xzf cilium-linux-${ARCH}.tar.gz -C /usr/local/bin --no-same-owner
HV=v1.20.4
U=https://github.com/cilium/hubble/releases/download/$HV
curl -sSLO $U/hubble-linux-${ARCH}.tar.gz
curl -sSLO $U/hubble-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c hubble-linux-${ARCH}.tar.gz.sha256sum
sudo tar xzf hubble-linux-${ARCH}.tar.gz -C /usr/local/bin --no-same-owner
source ~/.bashrc
helm version --short
cilium version --client
hubble versionExpected:
helm-v4.3.2-linux-amd64.tar.gz: OK
cilium-linux-amd64.tar.gz: OK
hubble-linux-amd64.tar.gz: OK
v4.3.2+g8346aaa
cilium-cli: v0.20.3 compiled with go1.27.3 on linux/amd64
cilium image (default): v1.20.3
cilium image (stable): v1.20.5
hubble v1.20.4@HEAD-3e268bb compiled with go1.27.3 on linux/amd64
On its own, the cilium CLI would install 1.20.3. That is why we install Cilium via Helm with a pinned version – the same way it runs in production:
helm repo add cilium https://helm.cilium.io/
helm repo update
helm search repo cilium/cilium --versions | head -3Expected:
NAME CHART VERSION APP VERSION DESCRIPTION
cilium/cilium 1.20.5 1.20.5 eBPF-based Networking, Security, and ...
cilium/cilium 1.20.4 1.20.4 eBPF-based Networking, Security, and ...
Save your progress? Sign up for free
Drill 6: Install Cilium and join the workers
Without kube-proxy, nobody knows the API server’s Service IP
10.96.0.1 yet. That is why Cilium gets its real address
(k8sServiceHost/k8sServicePort):
helm install cilium cilium/cilium --version 1.20.5 -n kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=172.16.0.126 --set k8sServicePort=6443 \
--set ipam.mode=cluster-pool \
--set ipam.operator.clusterPoolIPv4PodCIDRList=10.0.0.0/16 \
--set ipam.operator.clusterPoolIPv4MaskSize=24 \
--set routingMode=tunnel --set tunnelProtocol=vxlan \
--set hubble.enabled=true --set hubble.relay.enabled=true \
--set operator.replicas=1
kubeadm token create --print-join-commandExpected (abridged):
STATUS: deployed
...
You have successfully installed Cilium with Hubble Relay.
WARNINGS:
- WARNING: TLS is not enabled for the Hubble Relay server ...
Your release version is 1.20.5.
kubeadm join 172.16.0.126:6443 --token ... --discovery-token-ca-cert-hash sha256:...
Why 10.0.0.0/16? The chart default 10.0.0.0/8 would include the
Service network 10.96.0.0/12. A /16 with a /24 per node is enough for
256 nodes; cluster west gets 10.1.0.0/16 on Day 17. You
accept the TLS warning about Hubble Relay in the lab.
Run the printed kubeadm join line with sudo
on worker1 and worker2. Then back on
cp1:
cilium status --wait
k get nodes -o wide
k get ciliumnodes -o custom-columns=NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs
k get nodes -o custom-columns=NAME:.metadata.name,CIDR:.spec.podCIDRExpected (abridged):
Cilium: OK
Operator: OK
Envoy DaemonSet: OK
Hubble Relay: OK
ClusterMesh: disabled
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
...
NAME STATUS ROLES AGE VERSION INTERNAL-IP ...
cp1 Ready control-plane 7m53s v1.37.2 172.16.0.126 ...
worker1 Ready <none> 2m1s v1.37.2 172.16.0.228 ...
worker2 Ready <none> 109s v1.37.2 172.16.0.83 ...
NAME PODCIDRS
cp1 [10.0.0.0/24]
worker1 [10.0.1.0/24]
worker2 [10.0.2.0/24]
NAME CIDR
cp1 <none>
worker1 <none>
worker2 <none>
The Pod networks live in the CiliumNode objects, not on
the node – exactly as intended.
Warning:
cilium status --waitright afterhelm install, while onlycp1is in the cluster, runs into a timeout after 5 minutes: Hubble Relay does not tolerate the control plane taint and staysPending. Join the workers first, then wait.
SSH from cp1 to the other VMs
Later days jump from cp1 via ssh to the
workers and to router. On cp1, a lab key
without a passphrase:
ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519 -C ubuntu@cp1Expected (abridged):
Your identification has been saved in /home/ubuntu/.ssh/id_ed25519
Your public key has been saved in /home/ubuntu/.ssh/id_ed25519.pub
ssh-copy-id does not help because the VMs do not allow
password login. Your own computer, which reaches all VMs by key,
distributes the public key:
ssh [email protected] cat .ssh/id_ed25519.pub \
| ssh [email protected] 'cat >> .ssh/authorized_keys'
ssh [email protected] cat .ssh/id_ed25519.pub \
| ssh [email protected] 'cat >> .ssh/authorized_keys'
ssh [email protected] cat .ssh/id_ed25519.pub \
| ssh [email protected] 'cat >> .ssh/authorized_keys'Back on cp1. accept-new stores the host key
on first contact without asking:
ssh -o StrictHostKeyChecking=accept-new [email protected] hostname
ssh -o StrictHostKeyChecking=accept-new [email protected] hostname
ssh -o StrictHostKeyChecking=accept-new [email protected] hostnameExpected:
Warning: Permanently added '172.16.0.228' (ED25519) to the list of known hosts.
worker1
Warning: Permanently added '172.16.0.83' (ED25519) to the list of known hosts.
worker2
Warning: Permanently added '172.16.0.123' (ED25519) to the list of known hosts.
prod
The VM for the role router is called prod
in the lab – the book always says “router”. Today it only gets
up-to-date package lists; Day 15 installs FRR:
ssh [email protected] 'sudo apt-get update -qq && lsb_release -ds'Expected:
Ubuntu 24.04.5 LTS
From now on, all commands in the book run on cp1 unless
stated otherwise.
Save your progress? Sign up for free
Drill 7: Prove the kube-proxy replacement
KPR means: Cilium translates Service IPs and NodePorts with eBPF
directly in the kernel – at a Pod’s connect() (socket LB)
and at the network interface for traffic from outside. There are no more
iptables chains per Service. Three pieces of evidence:
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep -E 'KubeProxyReplacement|IPAM|Routing|Hubble:'
k get pods -A -l k8s-app=kube-proxy
sudo iptables-save | grep -c KUBE-SVC
ssh [email protected] 'sudo iptables-save | grep -c KUBE-SVC'Expected (abridged):
KubeProxyReplacement: True [eth0 172.16.0.126 ... (Direct Routing)]
IPAM: IPv4: 4/254 allocated from 10.0.0.0/24,
Routing: Network: Tunnel [vxlan] Host: Legacy
Hubble: Ok Current/Max Flows: 3182/4095 (77.70%), ...
No resources found
0
0
True plus the device eth0 means: KPR is
active and attached to this interface. With kube-proxy,
grep would find dozens of KUBE-SVC-… chains
here (Day 7 compares both).
The verbose output shows which functions KPR takes over:
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
| grep -A16 'KubeProxyReplacement Details'Expected (abridged):
KubeProxyReplacement Details:
Status: True
Socket LB: Enabled
...
Mode: SNAT
Backend Selection: Random
...
- ClusterIP: Enabled
- NodePort: Enabled (Range: 30000-32767)
- LoadBalancer: Enabled
Test Hubble: the hubble CLI talks to Hubble Relay, which runs in the
cluster. -P (--port-forward) sets up a port
forward to the relay for exactly this call and tears it down afterwards.
--port-forward-port 0 picks a free local port for it – so
parallel calls (such as a hubble observe -f in a second
terminal) do not get in each other’s way. The whole book uses this form;
a port forward in the background is not needed:
hubble status -P --port-forward-port 0
hubble list nodes -P --port-forward-port 0Expected (abridged, the local port changes with every call):
Healthcheck (via 127.0.0.1:36225): Ok
Current/Max Flows: 3,967/12,285 (32.29%)
Connected Nodes: 3/3
NAME STATUS AGE FLOWS/S CURRENT/MAX-FLOWS
cp1 Connected 4m44s 11.73 3369/4095 ( 82.27%)
worker1 Connected 1m10s 5.03 376/4095 ( 9.18%)
worker2 Connected 1m5s 3.17 226/4095 ( 5.52%)
Finally, Cilium’s big self-test. It creates its own namespaces, checks a good 80 scenarios (Pod-to-Pod, Services, policies, DNS) and takes about 20 minutes:
cilium connectivity testExpected (abridged, without the CLI’s symbols, may differ depending on the network):
[kubernetes] Creating namespace cilium-test-1 for connectivity check...
...
[cilium-test-1] Running 138 tests ...
...
Test Report [cilium-test-1]
1/82 tests failed (3/708 actions), 56 tests skipped, 0 scenarios skipped:
Test [no-unexpected-packet-drops]:
no-unexpected-packet-drops/no-unexpected-packet-drops:kubernetes/worker2: ...
"reason": "VLAN traffic disallowed by VLAN filter"
In the author’s lab exactly one test failed: the router in the home
network sends VLAN-tagged broadcasts (mDNS) to all VMs, and Cilium drops
foreign VLAN frames at eth0. That is not a fault in the
cluster. Whether it is the same for you is shown by
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg monitor --type drop
(stop with Ctrl+C). All other tests must be green. Then clean up:
k delete ns cilium-test-1 cilium-test-ccnp1 cilium-test-ccnp2Tip: The test looks for Hubble Relay hard-coded at
localhost:4245. Without a port forward in the background, it skips the flow checks (“Unable to contact Hubble Relay”) – hence the many skipped tests. That is enough for the functional test. If you want the flow checks, startcilium hubble port-forward &beforehand and stop it afterwards withpkill -f '^cilium hubble port-forward'– the only place in the book with a port forward in the background.
Save your progress? Sign up for free
Drill 8: Add-ons and notes-api
In the lab, metrics-server needs --kubelet-insecure-tls
because the kubelet certificates are self-signed. local-path becomes the
default StorageClass:
M=https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.1
k apply -f $M/components.yaml
k -n kube-system patch deployment metrics-server --type=json -p \
'[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
"value":"--kubelet-insecure-tls"}]'
L=https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37
k apply -f $L/deploy/local-path-storage.yaml
k patch storageclass local-path -p \
'{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
k get storageclassExpected (abridged):
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer
Now the sample application. NOTES_API_KEY is mandatory –
without it, notes-api does not start. That is why the variable goes into
the manifest before it reaches the cluster:
cd ~/ckne/tag00
k create namespace tag00
k -n tag00 create deployment notes-api \
--image=ghcr.io/thomaszachmann/notes-api:0.5.1 --replicas=2 --port=8080 \
$do > notes-api.yaml
k set env --local -f notes-api.yaml NOTES_API_KEY=lab-key-123 -o yaml \
> notes-api-env.yaml
k apply -f notes-api-env.yaml
k -n tag00 expose deployment notes-api --type=NodePort --port=80 --target-port=8080
k -n tag00 rollout status deployment notes-api
k -n tag00 get pods,svc -o wideExpected (abridged):
deployment "notes-api" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE
pod/notes-api-665494b9df-2j5pr 1/1 Running 0 2s 10.0.2.78 worker2
pod/notes-api-665494b9df-2zhdp 1/1 Running 0 1s 10.0.1.1 worker1
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notes-api NodePort 10.104.115.17 <none> 80:31368/TCP 1s
The Pod IPs come from the /24 networks of their nodes. The NodePort is assigned randomly, so it goes into variables:
CIP=$(k -n tag00 get svc notes-api -o jsonpath='{.spec.clusterIP}')
NP=$(k -n tag00 get svc notes-api -o jsonpath='{.spec.ports[0].nodePort}')
curl -s http://$CIP/version
curl -s -X POST -H 'X-API-Key: lab-key-123' -d '{"text":"Tag 0 steht"}' \
http://$CIP/notes
curl -s -o /dev/null -w '%{http_code}\n' -X POST -d '{"text":"x"}' http://$CIP/notes
for ip in 172.16.0.126 172.16.0.228 172.16.0.83; do
curl -s -o /dev/null -w "$ip:$NP %{http_code}\n" http://$ip:$NP/healthz
done
ssh [email protected] "curl -s -o /dev/null -w '%{http_code}\n' \
http://172.16.0.126:$NP/healthz"Expected:
0.5.1
{"id":"3dd797ea-1fe4-4c2b-843c-dc729071b672","text":"Tag 0 steht"}
401
172.16.0.126:31368 200
172.16.0.228:31368 200
172.16.0.83:31368 200
200
Without a key, notes-api answers with 401. The NodePort answers on
all nodes – even on cp1, where no notes-api Pod runs at all
– and from router, that is, from outside the cluster.
Cilium alone takes care of that. Where it knows the targets from is
shown by its Service table (an eBPF map):
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 -E "$CIP|:$NP/"Expected:
23 0.0.0.0:31368/TCP NodePort 1 => 10.0.1.1:8080/TCP (active)
2 => 10.0.2.78:8080/TCP (active)
25 10.104.115.17:80/TCP ClusterIP 1 => 10.0.1.1:8080/TCP (active)
2 => 10.0.2.78:8080/TCP (active)
Save your progress? Sign up for free
Exercise: Speed Drills
- Show whether Cilium replaces kube-proxy. Target time: 20 s
- Show the Pod networks Cilium has assigned to the nodes. Target time: 20 s
- Show the Helm values Cilium was installed with. Target time: 10 s
- Show the latest flows to port 8080 in the namespace
tag00. Target time: 30 s
Solution
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep KubeProxyReplacement
k get ciliumnodes -o custom-columns=NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs
helm get values cilium -n kube-system
hubble observe -P --port-forward-port 0 -n tag00 --protocol tcp --to-port 8080 \
--last 3Expected (abridged):
KubeProxyReplacement: True [eth0 172.16.0.126 ... (Direct Routing)]
NAME PODCIDRS
cp1 [10.0.0.0/24]
worker1 [10.0.1.0/24]
worker2 [10.0.2.0/24]
USER-SUPPLIED VALUES:
hubble:
enabled: true
relay:
enabled: true
ipam:
mode: cluster-pool
...
kubeProxyReplacement: true
...
10.0.0.88:54736 (world) -> tag00/notes-api-...-2zhdp:8080 (ID:25180) to-endpoint ...
10.0.0.88:54736 (world) -> tag00/notes-api-...-2zhdp:8080 (ID:25180) to-endpoint ...
10.0.0.88:54736 (world) -> tag00/notes-api-...-2zhdp:8080 (ID:25180) to-overlay ...
(Timestamps before each flow line omitted here.)
10.0.0.88 is the cilium_host address of
cp1: your curl from the node goes through the
tunnel.
Save your progress? Sign up for free
Checkpoint
k get nodes -o wideExpected: three nodes Ready,
v1.37.2, containerd://2.4.2.
k get pods -A --field-selector=status.phase!=RunningExpected: No resources found
cilium statusExpected: Cilium: OK,
Operator: OK, Envoy DaemonSet: OK,
Hubble Relay: OK, DaemonSet cilium with
Ready: 3/3.
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep KubeProxyReplacementExpected:
KubeProxyReplacement: True [eth0 …]
sudo iptables-save | grep -c KUBE-SVCExpected: 0
helm list -AExpected: exactly one release cilium in
kube-system, chart cilium-1.20.5,
deployed.
k top nodesExpected: CPU and memory values for
cp1, worker1, worker2.
ssh [email protected] hostnameExpected: prod – without a prompt for a
password or host key.
Challenge
Find out which node runs the notes-api Pod with the IP from
10.0.2.0/24, and show its endpoint including its security
identity in the Cilium agent of that node.
Hint: --field-selector spec.nodeName=…
finds the matching Cilium Pod, cilium-dbg endpoint list the
endpoint.
Solution
k -n tag00 get pods -o wide
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
| grep -E 'ENDPOINT|notes-api'
k get ciliumendpoints -n tag00Expected (abridged):
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS ... IPv4
1887 Disabled Disabled 25180 k8s:app=notes-api 10.0.2.78
NAME SECURITY IDENTITY ENDPOINT STATE IPV4
notes-api-665494b9df-2j5pr 25180 ready 10.0.2.78
notes-api-665494b9df-2zhdp 25180 ready 10.0.1.1
cilium-dbg shows only the endpoints of its own node;
ds/cilium lands in some arbitrary agent. Both Pods share
identity 25180 because they carry the same labels – the policies from
Day 12 on build on that.
Save your progress? Sign up for free
Cleanup
k delete namespace tag00Cilium, metrics-server and local-path stay for all following days.
Exam Tip
- In the CKNE you do not build a cluster, but you have to recognize
who implements Services: kube-proxy Pods present?
cilium-dbg status | grep KubeProxyReplacement? cilium-dbgonly works in the agent of the affected node – pick the right Pod with--field-selector spec.nodeName=….- Docs to know: docs.cilium.io “Kubernetes Without kube-proxy” and kubernetes.io “Creating a cluster with kubeadm”.
Enterprise Note
- Production clusters come from managed offerings or distributions; many now ship Cilium (GKE Dataplane V2, AKS with Cilium, RKE2, Talos).
- Cilium runs via Helm with versioned values from Git (GitOps), upgrades after a pre-flight check (Upgrade Guide) and release notes.
- In production,
k8sServiceHostpoints to a load balancer or a VIP in front of several API servers, never to a single node IP. - Hubble Relay with TLS (
hubble.tls,hubble.relay.tls.server.enabled), metrics-server without--kubelet-insecure-tls. - Pod CIDRs are planned centrally: no overlap with the Service network, VPN, other clusters (ClusterMesh) or corporate networks.