Skip to content
Hands-on IT IT Drills

Day 1: Exam Craft

Goal: You know the format and curriculum of the CKNE, work fast in the terminal and use Linux tools and cilium-dbg to find out where a packet currently is. · Duration: approx. 75 min · Prerequisite: Day 0

Briefing

  • The CKNE is a proctored online exam: 2 hours, hands-on tasks in a Linux remote desktop environment with a terminal and a browser. No multiple choice.
  • Curriculum: Core Infrastructure and CNI 15 % · Service Networking and DNS 25 % · Advanced Traffic Management 20 % · Network Security and Policy 25 % · Observability 15 %.
  • CKA knowledge is assumed (formally not required). Which documentation pages are allowed in the browser is listed in the Linux Foundation’s “Important Instructions” – read them before your appointment; practice as if you only had kubernetes.io and --help.
  • Today: the shell, contexts and the networking tools you need on every later day. Curriculum reference: foundation for all domains, especially “Observability”.

Drill 1: Check the shell instead of retyping

Day 0 already put k, $do, $now and completion into ~/.bashrc. In the exam this is usually prepared as well – check first, then type:

type k
echo $do
echo $now
complete -p k
cat ~/.vimrc

Expected:

k is aliased to `kubectl'
--dry-run=client -o yaml
--force --grace-period=0
complete -o default -F __start_kubectl k
set expandtab
set tabstop=2
set shiftwidth=2

If something is missing, append only the missing lines from Day 0, Drill 4. $do generates YAML without creating anything (k create … $do > x.yaml), $now deletes without waiting (k delete pod x $now). Without .vimrc you get tabs in YAML – a broken manifest.

Save your progress? Sign up for free

Drill 2: Contexts and namespaces

Many tasks start with “Switch to the context …”. Skip that, and you solve the task in the wrong cluster.

mkdir -p ~/ckne/tag01 && cd ~/ckne/tag01
k config get-contexts
k create ns tag01
k config set-context tag01 --cluster=kubernetes --user=kubernetes-admin \
  --namespace=tag01
k config use-context tag01
k config current-context
k config view --minify -o jsonpath='{..namespace}'; echo

Expected:

CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin
namespace/tag01 created
Context "tag01" created.
Switched to context "tag01".
tag01
tag01

From now on, everything without -n lands in tag01. To change only the namespace of the current context, use k config set-context --current --namespace=…. Two test Pods for the rest of the day – nicolaka/netshoot is a toolbox (tcpdump, dig, curl, ss, iperf3) and is available for amd64 and arm64:

k run web --image=nginx:1.29 --port=80 -l app=web
k run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k wait --for=condition=Ready pod/web pod/client --timeout=120s
k get pods -o wide

Expected (abridged):

pod/web condition met
pod/client condition met
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE      ...
client   1/1     Running   0          47s   10.0.1.27    worker1   ...
web      1/1     Running   0          48s   10.0.2.248   worker2   ...

On your cluster the nodes may be swapped – use your IPs and nodes in the following drills.

Save your progress? Sign up for free

Drill 3: ip – interfaces, addresses, routes

ip is the most important tool on the node. -br (brief) prints one line per interface. On cp1:

ip -br link
ip -br addr show cilium_host
ip route
ip route get 10.0.2.248
ip neigh show dev eth0

Expected (abridged):

lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0             UP             bc:24:11:52:2f:dd <BROADCAST,MULTICAST,UP,LOWER_UP>
cilium_net@cilium_host UP       0a:37:51:d1:85:de <BROADCAST,MULTICAST,NOARP,UP,...>
cilium_host@cilium_net UP       42:99:2d:97:ac:63 <BROADCAST,MULTICAST,NOARP,UP,...>
cilium_vxlan     UNKNOWN        7e:dd:13:ce:5d:08 <BROADCAST,MULTICAST,UP,LOWER_UP>
lxc_health@if6   UP             0e:20:e9:ac:ef:6c <BROADCAST,MULTICAST,UP,LOWER_UP>
lxc19d4d27e6140@if8 UP          a6:95:5d:38:f7:2a <BROADCAST,MULTICAST,UP,LOWER_UP>
cilium_host@cilium_net UP       10.0.0.88/32 fe80::4099:2dff:fe97:ac63/64
default via 172.16.0.1 dev eth0 proto dhcp src 172.16.0.126 metric 100
10.0.0.0/24 via 10.0.0.88 dev cilium_host proto kernel src 10.0.0.88
10.0.0.88 dev cilium_host proto kernel scope link
10.0.1.0/24 via 10.0.0.88 dev cilium_host proto kernel src 10.0.0.88 mtu 1450
10.0.2.0/24 via 10.0.0.88 dev cilium_host proto kernel src 10.0.0.88 mtu 1450
172.16.0.0/24 dev eth0 proto kernel scope link src 172.16.0.126 metric 100
10.0.2.248 dev cilium_host src 10.0.0.88 uid 1000
    cache mtu 1450
172.16.0.228 lladdr bc:24:11:5d:96:ab REACHABLE
172.16.0.123 lladdr bc:24:11:4d:d9:08 REACHABLE

How to read this: cilium_host is the gateway of all Pods on this node, and the Pod networks of the other nodes point there as well – eBPF handles the rest via cilium_vxlan. mtu 1450: 50 bytes go to the VXLAN header. Every lxc… is the host side of a Pod veth, @if8 the interface number of its peer inside the Pod.

Save your progress? Sign up for free

Drill 4: ss – who listens, who talks

ss replaces netstat: -t TCP, -u UDP, -l listening, -n no names, -p process (needs sudo). Filters in parentheses save you a grep:

sudo ss -tlnp '( sport = :6443 or sport = :10250 or sport = :4240 or sport = :4244 )'
sudo ss -tn state established '( dport = :6443 )' | head -3

Expected (abridged):

State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN 0      4096    172.16.0.126:4240       0.0.0.0:*    users:(("cilium-agent",...))
LISTEN 0      4096               *:10250            *:*    users:(("kubelet",...))
LISTEN 0      4096               *:6443             *:*    users:(("kube-apiserver",..
LISTEN 0      4096               *:4244             *:*    users:(("cilium-agent",...))
Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
0      0       172.16.0.126:57462 172.16.0.126:6443
0      0       172.16.0.126:56990 172.16.0.126:6443

4240 is the health check between the Cilium agents, 4244 the Hubble server that Hubble Relay queries. If a service “doesn’t work”, ss -tlnp first checks whether it listens at all – and on which address (127.0.0.1 is unreachable from outside).

Save your progress? Sign up for free

Drill 5: iptables and nftables – almost empty thanks to KPR

With kube-proxy there would be hundreds of rules here. On cp1:

sudo iptables -V
sudo iptables-save | grep -c '^-A'
sudo iptables-save | grep -c KUBE-SVC
sudo iptables-save -t nat | grep '^-A CILIUM_POST_nat' | head -2
sudo nft list tables

Expected (abridged):

iptables v1.8.10 (nf_tables)
35
0
-A CILIUM_POST_nat -s 10.0.0.0/24 ! -d 10.0.0.0/24 ! -o cilium_+ -m comment
--comment "cilium masquerade non-cluster" -j MASQUERADE
-A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy
return traffic from masquerade" -j ACCEPT
table ip mangle
table ip filter
table ip6 mangle
table ip nat
table ip6 nat
table ip6 filter
table ip raw

35 rules, not a single KUBE-SVC chain: Cilium translates Services in eBPF. What remains is masquerading for outbound traffic (Day 7 and 16) and the kubelet’s KUBE-FIREWALL. (nf_tables) means: iptables is just a frontend for nftables here – nft list ruleset shows the same rules, with the note “managed by iptables-nft, do not touch!”.

Tip: If a task asks “How are Services implemented?”, iptables-save | grep -c KUBE-SVC and k -n kube-system get ds are the fastest clues.

Save your progress? Sign up for free

Drill 6: tcpdump on the node

The most important switches: -i any all interfaces, -nn no host or port names (faster, unambiguous), -c N stop after N packets, followed by a filter (host, port, tcp, udp, and, or, not). The web Pod runs on worker2 – listen there:

ssh [email protected]
sudo tcpdump -i any -nn -c 4 host 10.0.2.248 and tcp port 80

In a second terminal on cp1:

k exec client -- curl -s -o /dev/null -w '%{http_code}\n' http://10.0.2.248

Expected (abridged, on worker2):

listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
23:18:21.547215 cilium_vxlan P   IP 10.0.1.27.40446 > 10.0.2.248.80: Flags [S], ...
23:18:21.547368 lxc451ed71841b8 Out IP 10.0.1.27.40446 > 10.0.2.248.80: Flags [S], ...
23:18:21.547412 lxc451ed71841b8 In  IP 10.0.2.248.80 > 10.0.1.27.40446: Flags [S.], ...
23:18:21.547427 cilium_vxlan Out IP 10.0.2.248.80 > 10.0.1.27.40446: Flags [S.], ...
4 packets captured

With -i any, each packet is prefixed with its interface: the SYN comes out of the tunnel already unwrapped (cilium_vxlan) and goes through the lxc interface into the Pod. From the host’s point of view, “into the Pod” is Out. Use exit to get back to cp1. Day 4 also looks at the packet on eth0, still encapsulated.

Save your progress? Sign up for free

Drill 7: nsenter and k debug – into a Pod’s network

The nginx image has neither ip nor ss. nsenter -n starts a host tool in the Pod’s network namespace. You know the way there from the CKA: Pod → container ID → PID. On worker2:

ssh [email protected]
sudo crictl pods --namespace tag01 --name web
ID=$(sudo crictl ps --name web -q --label io.kubernetes.pod.namespace=tag01)
P=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
echo $P
sudo nsenter -t $P -n ip -br addr
sudo nsenter -t $P -n ip route
sudo nsenter -t $P -n ss -tlnp

Expected (abridged):

POD ID          CREATED              STATE   NAME   NAMESPACE   ATTEMPT   RUNTIME
820ca44c235f8   About a minute ago   Ready   web    tag01       0         (default)
25308
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0@if33        UP             10.0.2.248/32 fe80::5018:bdff:fe0c:74eb/64
default via 10.0.2.241 dev eth0 mtu 1450
10.0.2.241 dev eth0 scope link
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      511          0.0.0.0:80        0.0.0.0:*    users:(("nginx",...))
LISTEN 0      511             [::]:80           [::]:*    users:(("nginx",...))

The Pod has a /32 address and a default route to 10.0.2.241 – that is cilium_host on worker2. eth0@if33 reveals the peer on the host: interface number 33.

ip -o link | awk -F': ' '$1==33{print $2}'
sudo nsenter -t $P -n tcpdump -i eth0 -nn -c 2 icmp
exit

Expected (abridged, after k exec client -- ping -c 2 10.0.2.248 from a second terminal on cp1):

lxc451ed71841b8@if32
23:18:51.458515 IP 10.0.1.27 > 10.0.2.248: ICMP echo request, id 30, seq 1, length 64
23:18:51.458630 IP 10.0.2.248 > 10.0.1.27: ICMP echo reply, id 30, seq 1, length 64

tcpdump in the Pod network sees only the traffic of this one Pod – ideal when the node is busy.

A second way, without SSH to the node: k debug attaches an ephemeral container to the running Pod. It shares the Pod’s network, --target additionally its processes, and --profile=netadmin gives it NET_ADMIN and NET_RAW. Back on cp1:

k debug web -it --image=nicolaka/netshoot:v0.16 --target=web --profile=netadmin

In the debug container’s shell:

ss -tlnp
tcpdump -i eth0 -nn -c 2 icmp
exit

Expected (abridged, ping as above from a second terminal):

Targeting container "web". If you don't see processes from this container it may be
because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-tnspq.
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      511          0.0.0.0:80        0.0.0.0:*    users:(("nginx",pid=1,...
23:18:51.458515 IP 10.0.1.27 > 10.0.2.248: ICMP echo request, id 30, seq 1, length 64
23:18:51.458630 IP 10.0.2.248 > 10.0.1.27: ICMP echo reply, id 30, seq 1, length 64

An ephemeral container cannot be removed – it stays Terminated in the Pod until the Pod is deleted. The same works for a node: k debug node/worker2 starts a Pod in the node’s host network (host file system under /host):

k debug node/worker2 -it --image=nicolaka/netshoot:v0.16 --profile=netadmin
ip -br addr show cilium_host
exit
k get pods
k delete pod node-debugger-worker2-tpfh2 $now

Expected (abridged):

Creating debugging pod node-debugger-worker2-tpfh2 with container debugger on node
worker2.
cilium_host@cilium_net UP             10.0.2.241/32 fe80::501c:faff:fe6a:b5db/64
NAME                          READY   STATUS      RESTARTS   AGE
client                        1/1     Running     0          2m23s
node-debugger-worker2-tpfh2   0/1     Completed   0          5s
web                           1/1     Running     0          2m24s

The node debug Pod stays behind as Completed after exit – always clean up (take the name from k get pods).

Save your progress? Sign up for free

Drill 8: cilium-dbg in the right agent

cilium-dbg is the debug CLI inside the agent container. It only knows its own node. So always pick the agent of the affected node first:

CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --brief
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
  | grep -E 'ENDPOINT|10.0.2.248'
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg identity get 5472
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg node list
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache get 10.0.1.27

Expected (abridged):

OK
ENDPOINT   POLICY (ingress)   ...   IDENTITY   LABELS        IPv4         STATUS
755        Disabled           ...   5472       k8s:app=web   10.0.2.248   ready
ID     LABELS
5472   k8s:app=web
       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=tag01
       k8s:io.cilium.k8s.policy.cluster=default
       k8s:io.cilium.k8s.policy.serviceaccount=default
       k8s:io.kubernetes.pod.namespace=tag01
Name      IPv4 Address   Endpoint CIDR   IPv6 Address   Endpoint CIDR   Source
cp1       172.16.0.126   10.0.0.0/24                                    custom-resource
worker1   172.16.0.228   10.0.1.0/24                                    custom-resource
worker2   172.16.0.83    10.0.2.0/24                                    local
10.0.1.27 maps to identity identity=11890 encryptkey=0 tunnelendpoint=172.16.0.228
flags=hastunnel

Take the identity number from endpoint list. The ipcache is Cilium’s address book: for every IP the identity and – for remote Pods – the node the tunnel leads to. Other subcommands you need in this book: service list, monitor --type drop, bpf lb list, policy get, config.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which process listens on port 4244 on cp1? Target time: 15 s
  2. Through which interface does a packet from cp1 to 10.0.1.27 leave the host? Target time: 10 s
  3. Show that there are no kube-proxy Service chains in iptables. Target time: 10 s
  4. Show the endpoint ID and identity of the Pod client in the responsible Cilium agent. Target time: 40 s

Solution

sudo ss -tlnp '( sport = :4244 )'
ip route get 10.0.1.27
sudo iptables-save | grep -c KUBE-SVC
N=$(k get pod client -o jsonpath='{.spec.nodeName}')
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=$N -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
  | grep -E 'ENDPOINT|10.0.1.27'

Expected (abridged):

LISTEN 0      4096               *:4244             *:*    users:(("cilium-agent",...))
10.0.1.27 dev cilium_host src 10.0.0.88 uid 1000
    cache mtu 1450
0
ENDPOINT   POLICY (ingress)   ...   IDENTITY   LABELS      IPv4        STATUS
1384       Disabled           ...   11890      k8s:io...   10.0.1.27   ready

The endpoint line shows only the first label; cilium-dbg identity get shows all of them.

Save your progress? Sign up for free

Checkpoint

type k

Expected: k is aliased to `kubectl'

k config current-context

Expected: tag01

k get pods -o wide

Expected: client and web Running, no node-debugger-… Pod left.

k get pod web -o jsonpath='{.spec.ephemeralContainers[*].name}'; echo

Expected: a name like debugger-tnspq – the ephemeral container from Drill 7.

sudo iptables-save | grep -c KUBE-SVC

Expected: 0

Challenge

Find out what the lxc interface of the Pod web is called on its node – using only k and cilium-dbg, without SSH – and then capture a curl on the node on exactly that interface.

Hint: cilium-dbg endpoint get <id> -o jsonpath=… returns a list; the field is called status.networking.interface-name.

Solution

CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint get 755 \
  -o jsonpath='{[0].status.networking.interface-name}{"\n"}'
ssh [email protected] sudo tcpdump -i lxc451ed71841b8 -nn -c 2 tcp port 80

In the second terminal: k exec client -- curl -s -o /dev/null http://10.0.2.248.

Expected (abridged):

lxc451ed71841b8
... IP 10.0.1.27.40446 > 10.0.2.248.80: Flags [S], ...
... IP 10.0.2.248.80 > 10.0.1.27.40446: Flags [S.], ...

Save your progress? Sign up for free

Cleanup

k config use-context kubernetes-admin@kubernetes
k config delete-context tag01
k delete ns tag01

Nothing changed cluster-wide today. The image nicolaka/netshoot:v0.16 is now cached on the nodes – Day 4 uses it again.

Exam Tip

  • Every task starts with the given context or ssh command; exit at the end, otherwise you work on the wrong host in the next task.
  • Always run cilium-dbg in the agent of the affected node (--field-selector spec.nodeName=…); ds/cilium hits a random agent.
  • Docs: kubernetes.io “Debug Running Pods” (kubectl debug, ephemeral containers) and docs.cilium.io “Troubleshooting” and “Command Cheatsheet”.

Enterprise Note

  • kubectl debug and nodes/proxy are powerful: allow ephemeral containers and node debug Pods only for operations teams, via RBAC and Pod Security Admission.
  • Captures on production nodes often contain plaintext (HTTP, DNS): only with a filter, a limited duration (-c, -G) and after approval.
  • Pull debug images like netshoot from your own registry, with a fixed version and a signature – not latest from Docker Hub.
  • Instead of tcpdump on every node: Hubble (Day 20) shows flows cluster-wide with Pod names and policy verdicts.