Skip to content
Hands-on IT IT Drills

Day 0: Training Ground

Goal: A kubeadm cluster v1.37.2 with three nodes runs without kube-proxy; Cilium takes over the Pod network, Services and Hubble. · Duration: approx. 90 min · Prerequisite: three VMs with Ubuntu 24.04, a fourth one for the role router

Briefing

The CKNE tests Kubernetes networking on ready-made clusters. Here you build your own, deliberately without kube-proxy: Cilium replaces it with eBPF (kube-proxy replacement, KPR for short). Cilium assigns Pod IPs itself from 10.0.0.0/16, and packets between nodes travel through a VXLAN tunnel. Curriculum reference: foundation for “CKNE: Core Infrastructure and CNI”.

  cp1 .126                 worker1 .228            worker2 .83
  control-plane, kubectl,  worker                  worker
  helm, cilium, hubble
  4 vCPU / 11 GB           2 vCPU / 3 GB           2 vCPU / 3 GB
        |________________________|_______________________|______ router .123
          172.16.0.0/24 · Pods 10.0.0.0/16 (Cilium, VXLAN)      (prod, external
          Services 10.96.0.0/12 · no kube-proxy                  client/server from
                                                                 Day 0, BGP from Day 15)

Replace the IPs with your own. There is no name resolution between the VMs, which is why IPs appear everywhere. Drills 1 to 3 run on cp1, worker1 and worker2, the rest on cp1.

Drill 1: Prepare the kernel

All downloads in this book query the architecture (dpkg --print-architecture). On a Mac with Apple Silicon (for example VMs via multipass) that is arm64 – the commands stay the same, only the outputs in the book show amd64.

printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<'EOF' | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sudo sysctl --system >/dev/null
sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables
swapon --show | wc -l

Expected:

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
0

If swapon shows anything: run sudo swapoff -a and comment out the swap line in /etc/fstab. Why ip_forward? Without forwarding in the kernel, no packet gets from a Pod beyond the node.

Save your progress? Sign up for free

Drill 2: containerd and runc

Both straight from the project, each with a checksum:

cd /tmp
V=2.4.2
U=https://github.com/containerd/containerd/releases/download/v$V
ARCH=$(dpkg --print-architecture)
curl -sSLO $U/containerd-$V-linux-${ARCH}.tar.gz
curl -sSLO $U/containerd-$V-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c containerd-$V-linux-${ARCH}.tar.gz.sha256sum
sudo tar Cxzf /usr/local containerd-$V-linux-${ARCH}.tar.gz
sudo mkdir -p /usr/local/lib/systemd/system
sudo curl -sSL -o /usr/local/lib/systemd/system/containerd.service \
  https://raw.githubusercontent.com/containerd/containerd/v$V/containerd.service
R=1.5.3
U=https://github.com/opencontainers/runc/releases/download/v$R
curl -sSLO $U/runc.${ARCH}
curl -sSLO $U/runc.sha256sum
grep " runc.${ARCH}$" runc.sha256sum | sha256sum -c
sudo install -m 755 runc.${ARCH} /usr/local/sbin/runc
runc --version | head -1

Expected:

containerd-2.4.2-linux-amd64.tar.gz: OK
runc.amd64: OK
runc version 1.5.3

Generate the configuration and switch to the systemd cgroup driver (like the kubelet):

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
grep -n SystemdCgroup /etc/containerd/config.toml
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
containerd --version

Expected:

89:            SystemdCgroup = true
containerd github.com/containerd/containerd/v2 v2.4.2 b7a25261e029...

Save your progress? Sign up for free

Drill 3: kubeadm, kubelet, kubectl in v1.37.2

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg jq
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
K=/etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=$K] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
apt-cache madison kubeadm | head -2
sudo apt-get install -y kubelet=1.37.2-1.1 kubeadm=1.37.2-1.1 kubectl=1.37.2-1.1 \
  cri-tools
sudo apt-mark hold kubelet kubeadm kubectl
printf 'runtime-endpoint: unix:///run/containerd/containerd.sock\n' \
  | sudo tee /etc/crictl.yaml
printf 'image-endpoint: unix:///run/containerd/containerd.sock\n' \
  | sudo tee -a /etc/crictl.yaml
kubeadm version -o short
sudo crictl version

Expected (abridged):

   kubeadm | 1.37.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb  Packages
   kubeadm | 1.37.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.37/deb  Packages
...
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
...
v1.37.2
RuntimeName:  containerd
RuntimeVersion:  v2.4.2

hold prevents an apt upgrade from changing the cluster version on the side. The package kubernetes-cni comes along as a dependency – it puts standard plugins into /opt/cni/bin that Cilium does not need (more on that on Day 2).

Save your progress? Sign up for free

Drill 4: Control plane without kube-proxy

On cp1. --skip-phases=addon/kube-proxy leaves out the kube-proxy DaemonSet. A --pod-network-cidr is deliberately missing: with IPAM cluster-pool, Cilium hands out the Pod networks itself, and the kube-controller-manager should not write a second, diverging allocation into node.spec.podCIDR.

sudo kubeadm init --kubernetes-version=v1.37.2 \
  --apiserver-advertise-address=172.16.0.126 --skip-phases=addon/kube-proxy
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
mkdir -p ~/ckne/tag00
kubectl get nodes
kubectl -n kube-system get ds
kubectl -n kube-system get cm kubeadm-config \
  -o jsonpath='{.data.ClusterConfiguration}' | grep -A1 '^proxy'

Expected (abridged):

Your Kubernetes control-plane has initialized successfully!
...
NAME   STATUS     ROLES           AGE   VERSION
cp1    NotReady   control-plane   64s   v1.37.2
No resources found in kube-system namespace.
proxy:
  disabled: true

NotReady is correct: without a CNI, no Pod gets an IP. No DaemonSet means: no kube-proxy. kubeadm remembers proxy.disabled: true so that kubeadm upgrade does not install it later either.

The basic exam setup for the shell on cp1 (completion for kubectl, cilium and hubble works from Drill 5 on):

cat <<'EOF' >> ~/.bashrc
source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k
export do="--dry-run=client -o yaml"
export now="--force --grace-period=0"
source <(cilium completion bash)
source <(hubble completion bash)
EOF
cat <<'EOF' >> ~/.vimrc
set expandtab
set tabstop=2
set shiftwidth=2
EOF

$do generates YAML without creating anything; $now deletes without waiting. The .vimrc ensures spaces instead of tabs in YAML. Day 1 goes deeper into this. Run source ~/.bashrc only after Drill 5, otherwise cilium and hubble are still missing.

Save your progress? Sign up for free

Drill 5: Helm, cilium CLI and hubble CLI

Three tools on cp1, all with a checksum. --no-same-owner makes sure the files are owned by root and not by the user ID from the archive:

cd /tmp
ARCH=$(dpkg --print-architecture)
H=v4.3.2
curl -sSLO https://get.helm.sh/helm-$H-linux-${ARCH}.tar.gz
curl -sSLO https://get.helm.sh/helm-$H-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c helm-$H-linux-${ARCH}.tar.gz.sha256sum
tar xzf helm-$H-linux-${ARCH}.tar.gz
sudo install -m 755 linux-${ARCH}/helm /usr/local/bin/helm
C=v0.20.3
U=https://github.com/cilium/cilium-cli/releases/download/$C
curl -sSLO $U/cilium-linux-${ARCH}.tar.gz
curl -sSLO $U/cilium-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c cilium-linux-${ARCH}.tar.gz.sha256sum
sudo tar xzf cilium-linux-${ARCH}.tar.gz -C /usr/local/bin --no-same-owner
HV=v1.20.4
U=https://github.com/cilium/hubble/releases/download/$HV
curl -sSLO $U/hubble-linux-${ARCH}.tar.gz
curl -sSLO $U/hubble-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c hubble-linux-${ARCH}.tar.gz.sha256sum
sudo tar xzf hubble-linux-${ARCH}.tar.gz -C /usr/local/bin --no-same-owner
source ~/.bashrc
helm version --short
cilium version --client
hubble version

Expected:

helm-v4.3.2-linux-amd64.tar.gz: OK
cilium-linux-amd64.tar.gz: OK
hubble-linux-amd64.tar.gz: OK
v4.3.2+g8346aaa
cilium-cli: v0.20.3 compiled with go1.27.3 on linux/amd64
cilium image (default): v1.20.3
cilium image (stable): v1.20.5
hubble v1.20.4@HEAD-3e268bb compiled with go1.27.3 on linux/amd64

On its own, the cilium CLI would install 1.20.3. That is why we install Cilium via Helm with a pinned version – the same way it runs in production:

helm repo add cilium https://helm.cilium.io/
helm repo update
helm search repo cilium/cilium --versions | head -3

Expected:

NAME            CHART VERSION   APP VERSION     DESCRIPTION
cilium/cilium   1.20.5          1.20.5          eBPF-based Networking, Security, and ...
cilium/cilium   1.20.4          1.20.4          eBPF-based Networking, Security, and ...

Save your progress? Sign up for free

Drill 6: Install Cilium and join the workers

Without kube-proxy, nobody knows the API server’s Service IP 10.96.0.1 yet. That is why Cilium gets its real address (k8sServiceHost/k8sServicePort):

helm install cilium cilium/cilium --version 1.20.5 -n kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=172.16.0.126 --set k8sServicePort=6443 \
  --set ipam.mode=cluster-pool \
  --set ipam.operator.clusterPoolIPv4PodCIDRList=10.0.0.0/16 \
  --set ipam.operator.clusterPoolIPv4MaskSize=24 \
  --set routingMode=tunnel --set tunnelProtocol=vxlan \
  --set hubble.enabled=true --set hubble.relay.enabled=true \
  --set operator.replicas=1
kubeadm token create --print-join-command

Expected (abridged):

STATUS: deployed
...
You have successfully installed Cilium with Hubble Relay.
WARNINGS:
- WARNING: TLS is not enabled for the Hubble Relay server ...
Your release version is 1.20.5.
kubeadm join 172.16.0.126:6443 --token ... --discovery-token-ca-cert-hash sha256:...

Why 10.0.0.0/16? The chart default 10.0.0.0/8 would include the Service network 10.96.0.0/12. A /16 with a /24 per node is enough for 256 nodes; cluster west gets 10.1.0.0/16 on Day 17. You accept the TLS warning about Hubble Relay in the lab.

Run the printed kubeadm join line with sudo on worker1 and worker2. Then back on cp1:

cilium status --wait
k get nodes -o wide
k get ciliumnodes -o custom-columns=NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs
k get nodes -o custom-columns=NAME:.metadata.name,CIDR:.spec.podCIDR

Expected (abridged):

    Cilium:             OK
    Operator:           OK
    Envoy DaemonSet:    OK
    Hubble Relay:       OK
    ClusterMesh:        disabled
DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
...
NAME      STATUS   ROLES           AGE     VERSION   INTERNAL-IP    ...
cp1       Ready    control-plane   7m53s   v1.37.2   172.16.0.126   ...
worker1   Ready    <none>          2m1s    v1.37.2   172.16.0.228   ...
worker2   Ready    <none>          109s    v1.37.2   172.16.0.83    ...
NAME      PODCIDRS
cp1       [10.0.0.0/24]
worker1   [10.0.1.0/24]
worker2   [10.0.2.0/24]
NAME      CIDR
cp1       <none>
worker1   <none>
worker2   <none>

The Pod networks live in the CiliumNode objects, not on the node – exactly as intended.

Warning: cilium status --wait right after helm install, while only cp1 is in the cluster, runs into a timeout after 5 minutes: Hubble Relay does not tolerate the control plane taint and stays Pending. Join the workers first, then wait.

SSH from cp1 to the other VMs

Later days jump from cp1 via ssh to the workers and to router. On cp1, a lab key without a passphrase:

ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519 -C ubuntu@cp1

Expected (abridged):

Your identification has been saved in /home/ubuntu/.ssh/id_ed25519
Your public key has been saved in /home/ubuntu/.ssh/id_ed25519.pub

ssh-copy-id does not help because the VMs do not allow password login. Your own computer, which reaches all VMs by key, distributes the public key:

ssh [email protected] cat .ssh/id_ed25519.pub \
  | ssh [email protected] 'cat >> .ssh/authorized_keys'
ssh [email protected] cat .ssh/id_ed25519.pub \
  | ssh [email protected] 'cat >> .ssh/authorized_keys'
ssh [email protected] cat .ssh/id_ed25519.pub \
  | ssh [email protected] 'cat >> .ssh/authorized_keys'

Back on cp1. accept-new stores the host key on first contact without asking:

ssh -o StrictHostKeyChecking=accept-new [email protected] hostname
ssh -o StrictHostKeyChecking=accept-new [email protected] hostname
ssh -o StrictHostKeyChecking=accept-new [email protected] hostname

Expected:

Warning: Permanently added '172.16.0.228' (ED25519) to the list of known hosts.
worker1
Warning: Permanently added '172.16.0.83' (ED25519) to the list of known hosts.
worker2
Warning: Permanently added '172.16.0.123' (ED25519) to the list of known hosts.
prod

The VM for the role router is called prod in the lab – the book always says “router”. Today it only gets up-to-date package lists; Day 15 installs FRR:

ssh [email protected] 'sudo apt-get update -qq && lsb_release -ds'

Expected:

Ubuntu 24.04.5 LTS

From now on, all commands in the book run on cp1 unless stated otherwise.

Save your progress? Sign up for free

Drill 7: Prove the kube-proxy replacement

KPR means: Cilium translates Service IPs and NodePorts with eBPF directly in the kernel – at a Pod’s connect() (socket LB) and at the network interface for traffic from outside. There are no more iptables chains per Service. Three pieces of evidence:

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep -E 'KubeProxyReplacement|IPAM|Routing|Hubble:'
k get pods -A -l k8s-app=kube-proxy
sudo iptables-save | grep -c KUBE-SVC
ssh [email protected] 'sudo iptables-save | grep -c KUBE-SVC'

Expected (abridged):

KubeProxyReplacement:    True   [eth0    172.16.0.126 ... (Direct Routing)]
IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24,
Routing:                 Network: Tunnel [vxlan]   Host: Legacy
Hubble:                  Ok              Current/Max Flows: 3182/4095 (77.70%), ...
No resources found
0
0

True plus the device eth0 means: KPR is active and attached to this interface. With kube-proxy, grep would find dozens of KUBE-SVC-… chains here (Day 7 compares both).

The verbose output shows which functions KPR takes over:

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
  | grep -A16 'KubeProxyReplacement Details'

Expected (abridged):

KubeProxyReplacement Details:
  Status:               True
  Socket LB:            Enabled
...
  Mode:                 SNAT
  Backend Selection:    Random
...
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767)
  - LoadBalancer:   Enabled

Test Hubble: the hubble CLI talks to Hubble Relay, which runs in the cluster. -P (--port-forward) sets up a port forward to the relay for exactly this call and tears it down afterwards. --port-forward-port 0 picks a free local port for it – so parallel calls (such as a hubble observe -f in a second terminal) do not get in each other’s way. The whole book uses this form; a port forward in the background is not needed:

hubble status -P --port-forward-port 0
hubble list nodes -P --port-forward-port 0

Expected (abridged, the local port changes with every call):

Healthcheck (via 127.0.0.1:36225): Ok
Current/Max Flows: 3,967/12,285 (32.29%)
Connected Nodes: 3/3
NAME      STATUS      AGE     FLOWS/S   CURRENT/MAX-FLOWS
cp1       Connected   4m44s   11.73     3369/4095 ( 82.27%)
worker1   Connected   1m10s   5.03      376/4095 (  9.18%)
worker2   Connected   1m5s    3.17      226/4095 (  5.52%)

Finally, Cilium’s big self-test. It creates its own namespaces, checks a good 80 scenarios (Pod-to-Pod, Services, policies, DNS) and takes about 20 minutes:

cilium connectivity test

Expected (abridged, without the CLI’s symbols, may differ depending on the network):

[kubernetes] Creating namespace cilium-test-1 for connectivity check...
...
[cilium-test-1] Running 138 tests ...
...
Test Report [cilium-test-1]
1/82 tests failed (3/708 actions), 56 tests skipped, 0 scenarios skipped:
Test [no-unexpected-packet-drops]:
  no-unexpected-packet-drops/no-unexpected-packet-drops:kubernetes/worker2: ...
    "reason": "VLAN traffic disallowed by VLAN filter"

In the author’s lab exactly one test failed: the router in the home network sends VLAN-tagged broadcasts (mDNS) to all VMs, and Cilium drops foreign VLAN frames at eth0. That is not a fault in the cluster. Whether it is the same for you is shown by k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg monitor --type drop (stop with Ctrl+C). All other tests must be green. Then clean up:

k delete ns cilium-test-1 cilium-test-ccnp1 cilium-test-ccnp2

Tip: The test looks for Hubble Relay hard-coded at localhost:4245. Without a port forward in the background, it skips the flow checks (“Unable to contact Hubble Relay”) – hence the many skipped tests. That is enough for the functional test. If you want the flow checks, start cilium hubble port-forward & beforehand and stop it afterwards with pkill -f '^cilium hubble port-forward' – the only place in the book with a port forward in the background.

Save your progress? Sign up for free

Drill 8: Add-ons and notes-api

In the lab, metrics-server needs --kubelet-insecure-tls because the kubelet certificates are self-signed. local-path becomes the default StorageClass:

M=https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.1
k apply -f $M/components.yaml
k -n kube-system patch deployment metrics-server --type=json -p \
  '[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
     "value":"--kubelet-insecure-tls"}]'
L=https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37
k apply -f $L/deploy/local-path-storage.yaml
k patch storageclass local-path -p \
  '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
k get storageclass

Expected (abridged):

NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer

Now the sample application. NOTES_API_KEY is mandatory – without it, notes-api does not start. That is why the variable goes into the manifest before it reaches the cluster:

cd ~/ckne/tag00
k create namespace tag00
k -n tag00 create deployment notes-api \
  --image=ghcr.io/thomaszachmann/notes-api:0.5.1 --replicas=2 --port=8080 \
  $do > notes-api.yaml
k set env --local -f notes-api.yaml NOTES_API_KEY=lab-key-123 -o yaml \
  > notes-api-env.yaml
k apply -f notes-api-env.yaml
k -n tag00 expose deployment notes-api --type=NodePort --port=80 --target-port=8080
k -n tag00 rollout status deployment notes-api
k -n tag00 get pods,svc -o wide

Expected (abridged):

deployment "notes-api" successfully rolled out
NAME                             READY   STATUS    RESTARTS   AGE   IP          NODE
pod/notes-api-665494b9df-2j5pr   1/1     Running   0          2s    10.0.2.78   worker2
pod/notes-api-665494b9df-2zhdp   1/1     Running   0          1s    10.0.1.1    worker1
NAME                TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/notes-api   NodePort   10.104.115.17   <none>        80:31368/TCP   1s

The Pod IPs come from the /24 networks of their nodes. The NodePort is assigned randomly, so it goes into variables:

CIP=$(k -n tag00 get svc notes-api -o jsonpath='{.spec.clusterIP}')
NP=$(k -n tag00 get svc notes-api -o jsonpath='{.spec.ports[0].nodePort}')
curl -s http://$CIP/version
curl -s -X POST -H 'X-API-Key: lab-key-123' -d '{"text":"Tag 0 steht"}' \
  http://$CIP/notes
curl -s -o /dev/null -w '%{http_code}\n' -X POST -d '{"text":"x"}' http://$CIP/notes
for ip in 172.16.0.126 172.16.0.228 172.16.0.83; do
  curl -s -o /dev/null -w "$ip:$NP %{http_code}\n" http://$ip:$NP/healthz
done
ssh [email protected] "curl -s -o /dev/null -w '%{http_code}\n' \
  http://172.16.0.126:$NP/healthz"

Expected:

0.5.1
{"id":"3dd797ea-1fe4-4c2b-843c-dc729071b672","text":"Tag 0 steht"}
401
172.16.0.126:31368 200
172.16.0.228:31368 200
172.16.0.83:31368 200
200

Without a key, notes-api answers with 401. The NodePort answers on all nodes – even on cp1, where no notes-api Pod runs at all – and from router, that is, from outside the cluster. Cilium alone takes care of that. Where it knows the targets from is shown by its Service table (an eBPF map):

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 -E "$CIP|:$NP/"

Expected:

23   0.0.0.0:31368/TCP       NodePort       1 => 10.0.1.1:8080/TCP (active)
                                            2 => 10.0.2.78:8080/TCP (active)
25   10.104.115.17:80/TCP    ClusterIP      1 => 10.0.1.1:8080/TCP (active)
                                            2 => 10.0.2.78:8080/TCP (active)

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show whether Cilium replaces kube-proxy. Target time: 20 s
  2. Show the Pod networks Cilium has assigned to the nodes. Target time: 20 s
  3. Show the Helm values Cilium was installed with. Target time: 10 s
  4. Show the latest flows to port 8080 in the namespace tag00. Target time: 30 s

Solution

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep KubeProxyReplacement
k get ciliumnodes -o custom-columns=NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs
helm get values cilium -n kube-system
hubble observe -P --port-forward-port 0 -n tag00 --protocol tcp --to-port 8080 \
  --last 3

Expected (abridged):

KubeProxyReplacement:    True   [eth0    172.16.0.126 ... (Direct Routing)]
NAME      PODCIDRS
cp1       [10.0.0.0/24]
worker1   [10.0.1.0/24]
worker2   [10.0.2.0/24]
USER-SUPPLIED VALUES:
hubble:
  enabled: true
  relay:
    enabled: true
ipam:
  mode: cluster-pool
...
kubeProxyReplacement: true
...
10.0.0.88:54736 (world) -> tag00/notes-api-...-2zhdp:8080 (ID:25180) to-endpoint ...
10.0.0.88:54736 (world) -> tag00/notes-api-...-2zhdp:8080 (ID:25180) to-endpoint ...
10.0.0.88:54736 (world) -> tag00/notes-api-...-2zhdp:8080 (ID:25180) to-overlay ...

(Timestamps before each flow line omitted here.) 10.0.0.88 is the cilium_host address of cp1: your curl from the node goes through the tunnel.

Save your progress? Sign up for free

Checkpoint

k get nodes -o wide

Expected: three nodes Ready, v1.37.2, containerd://2.4.2.

k get pods -A --field-selector=status.phase!=Running

Expected: No resources found

cilium status

Expected: Cilium: OK, Operator: OK, Envoy DaemonSet: OK, Hubble Relay: OK, DaemonSet cilium with Ready: 3/3.

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep KubeProxyReplacement

Expected: KubeProxyReplacement: True [eth0 …]

sudo iptables-save | grep -c KUBE-SVC

Expected: 0

helm list -A

Expected: exactly one release cilium in kube-system, chart cilium-1.20.5, deployed.

k top nodes

Expected: CPU and memory values for cp1, worker1, worker2.

ssh [email protected] hostname

Expected: prod – without a prompt for a password or host key.

Challenge

Find out which node runs the notes-api Pod with the IP from 10.0.2.0/24, and show its endpoint including its security identity in the Cilium agent of that node.

Hint: --field-selector spec.nodeName=… finds the matching Cilium Pod, cilium-dbg endpoint list the endpoint.

Solution

k -n tag00 get pods -o wide
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
  | grep -E 'ENDPOINT|notes-api'
k get ciliumendpoints -n tag00

Expected (abridged):

ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS ...   IPv4
1887       Disabled           Disabled          25180      k8s:app=notes-api   10.0.2.78
NAME                         SECURITY IDENTITY   ENDPOINT STATE   IPV4
notes-api-665494b9df-2j5pr   25180               ready            10.0.2.78
notes-api-665494b9df-2zhdp   25180               ready            10.0.1.1

cilium-dbg shows only the endpoints of its own node; ds/cilium lands in some arbitrary agent. Both Pods share identity 25180 because they carry the same labels – the policies from Day 12 on build on that.

Save your progress? Sign up for free

Cleanup

k delete namespace tag00

Cilium, metrics-server and local-path stay for all following days.

Exam Tip

  • In the CKNE you do not build a cluster, but you have to recognize who implements Services: kube-proxy Pods present? cilium-dbg status | grep KubeProxyReplacement?
  • cilium-dbg only works in the agent of the affected node – pick the right Pod with --field-selector spec.nodeName=….
  • Docs to know: docs.cilium.io “Kubernetes Without kube-proxy” and kubernetes.io “Creating a cluster with kubeadm”.

Enterprise Note

  • Production clusters come from managed offerings or distributions; many now ship Cilium (GKE Dataplane V2, AKS with Cilium, RKE2, Talos).
  • Cilium runs via Helm with versioned values from Git (GitOps), upgrades after a pre-flight check (Upgrade Guide) and release notes.
  • In production, k8sServiceHost points to a load balancer or a VIP in front of several API servers, never to a single node IP.
  • Hubble Relay with TLS (hubble.tls, hubble.relay.tls.server.enabled), metrics-server without --kubelet-insecure-tls.
  • Pod CIDRs are planned centrally: no overlap with the Service network, VPN, other clusters (ClusterMesh) or corporate networks.