Day 1: Exam Craft
Goal: You know the format and curriculum of the CKNE, work fast in the terminal and use Linux tools and
cilium-dbgto find out where a packet currently is. · Duration: approx. 75 min · Prerequisite: Day 0
Briefing
- The CKNE is a proctored online exam: 2 hours, hands-on tasks in a Linux remote desktop environment with a terminal and a browser. No multiple choice.
- Curriculum: Core Infrastructure and CNI 15 % · Service Networking and DNS 25 % · Advanced Traffic Management 20 % · Network Security and Policy 25 % · Observability 15 %.
- CKA knowledge is assumed (formally not required). Which
documentation pages are allowed in the browser is listed in the Linux
Foundation’s “Important Instructions” – read them before your
appointment; practice as if you only had kubernetes.io and
--help. - Today: the shell, contexts and the networking tools you need on every later day. Curriculum reference: foundation for all domains, especially “Observability”.
Drill 1: Check the shell instead of retyping
Day 0 already put k, $do, $now
and completion into ~/.bashrc. In the exam this is usually
prepared as well – check first, then type:
type k
echo $do
echo $now
complete -p k
cat ~/.vimrcExpected:
k is aliased to `kubectl'
--dry-run=client -o yaml
--force --grace-period=0
complete -o default -F __start_kubectl k
set expandtab
set tabstop=2
set shiftwidth=2
If something is missing, append only the missing lines from Day 0,
Drill 4. $do generates YAML without creating anything
(k create … $do > x.yaml), $now deletes
without waiting (k delete pod x $now). Without
.vimrc you get tabs in YAML – a broken manifest.
Save your progress? Sign up for free
Drill 2: Contexts and namespaces
Many tasks start with “Switch to the context …”. Skip that, and you solve the task in the wrong cluster.
mkdir -p ~/ckne/tag01 && cd ~/ckne/tag01
k config get-contexts
k create ns tag01
k config set-context tag01 --cluster=kubernetes --user=kubernetes-admin \
--namespace=tag01
k config use-context tag01
k config current-context
k config view --minify -o jsonpath='{..namespace}'; echoExpected:
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* kubernetes-admin@kubernetes kubernetes kubernetes-admin
namespace/tag01 created
Context "tag01" created.
Switched to context "tag01".
tag01
tag01
From now on, everything without -n lands in
tag01. To change only the namespace of the current context,
use k config set-context --current --namespace=…. Two test
Pods for the rest of the day – nicolaka/netshoot is a
toolbox (tcpdump, dig, curl, ss, iperf3) and is available for amd64 and
arm64:
k run web --image=nginx:1.29 --port=80 -l app=web
k run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k wait --for=condition=Ready pod/web pod/client --timeout=120s
k get pods -o wideExpected (abridged):
pod/web condition met
pod/client condition met
NAME READY STATUS RESTARTS AGE IP NODE ...
client 1/1 Running 0 47s 10.0.1.27 worker1 ...
web 1/1 Running 0 48s 10.0.2.248 worker2 ...
On your cluster the nodes may be swapped – use your IPs and nodes in the following drills.
Save your progress? Sign up for free
Drill 3: ip – interfaces, addresses, routes
ip is the most important tool on the node.
-br (brief) prints one line per interface. On
cp1:
ip -br link
ip -br addr show cilium_host
ip route
ip route get 10.0.2.248
ip neigh show dev eth0Expected (abridged):
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0 UP bc:24:11:52:2f:dd <BROADCAST,MULTICAST,UP,LOWER_UP>
cilium_net@cilium_host UP 0a:37:51:d1:85:de <BROADCAST,MULTICAST,NOARP,UP,...>
cilium_host@cilium_net UP 42:99:2d:97:ac:63 <BROADCAST,MULTICAST,NOARP,UP,...>
cilium_vxlan UNKNOWN 7e:dd:13:ce:5d:08 <BROADCAST,MULTICAST,UP,LOWER_UP>
lxc_health@if6 UP 0e:20:e9:ac:ef:6c <BROADCAST,MULTICAST,UP,LOWER_UP>
lxc19d4d27e6140@if8 UP a6:95:5d:38:f7:2a <BROADCAST,MULTICAST,UP,LOWER_UP>
cilium_host@cilium_net UP 10.0.0.88/32 fe80::4099:2dff:fe97:ac63/64
default via 172.16.0.1 dev eth0 proto dhcp src 172.16.0.126 metric 100
10.0.0.0/24 via 10.0.0.88 dev cilium_host proto kernel src 10.0.0.88
10.0.0.88 dev cilium_host proto kernel scope link
10.0.1.0/24 via 10.0.0.88 dev cilium_host proto kernel src 10.0.0.88 mtu 1450
10.0.2.0/24 via 10.0.0.88 dev cilium_host proto kernel src 10.0.0.88 mtu 1450
172.16.0.0/24 dev eth0 proto kernel scope link src 172.16.0.126 metric 100
10.0.2.248 dev cilium_host src 10.0.0.88 uid 1000
cache mtu 1450
172.16.0.228 lladdr bc:24:11:5d:96:ab REACHABLE
172.16.0.123 lladdr bc:24:11:4d:d9:08 REACHABLE
How to read this: cilium_host is the gateway of all Pods
on this node, and the Pod networks of the other nodes point there as
well – eBPF handles the rest via cilium_vxlan.
mtu 1450: 50 bytes go to the VXLAN header. Every
lxc… is the host side of a Pod veth, @if8 the
interface number of its peer inside the Pod.
Save your progress? Sign up for free
Drill 4: ss – who listens, who talks
ss replaces netstat: -t TCP,
-u UDP, -l listening, -n no
names, -p process (needs sudo). Filters in
parentheses save you a grep:
sudo ss -tlnp '( sport = :6443 or sport = :10250 or sport = :4240 or sport = :4244 )'
sudo ss -tn state established '( dport = :6443 )' | head -3Expected (abridged):
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 4096 172.16.0.126:4240 0.0.0.0:* users:(("cilium-agent",...))
LISTEN 0 4096 *:10250 *:* users:(("kubelet",...))
LISTEN 0 4096 *:6443 *:* users:(("kube-apiserver",..
LISTEN 0 4096 *:4244 *:* users:(("cilium-agent",...))
Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
0 0 172.16.0.126:57462 172.16.0.126:6443
0 0 172.16.0.126:56990 172.16.0.126:6443
4240 is the health check between the Cilium agents, 4244 the Hubble
server that Hubble Relay queries. If a service “doesn’t work”,
ss -tlnp first checks whether it listens at all – and on
which address (127.0.0.1 is unreachable from outside).
Save your progress? Sign up for free
Drill 5: iptables and nftables – almost empty thanks to KPR
With kube-proxy there would be hundreds of rules here. On
cp1:
sudo iptables -V
sudo iptables-save | grep -c '^-A'
sudo iptables-save | grep -c KUBE-SVC
sudo iptables-save -t nat | grep '^-A CILIUM_POST_nat' | head -2
sudo nft list tablesExpected (abridged):
iptables v1.8.10 (nf_tables)
35
0
-A CILIUM_POST_nat -s 10.0.0.0/24 ! -d 10.0.0.0/24 ! -o cilium_+ -m comment
--comment "cilium masquerade non-cluster" -j MASQUERADE
-A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy
return traffic from masquerade" -j ACCEPT
table ip mangle
table ip filter
table ip6 mangle
table ip nat
table ip6 nat
table ip6 filter
table ip raw
35 rules, not a single KUBE-SVC chain: Cilium translates
Services in eBPF. What remains is masquerading for outbound traffic (Day
7 and 16) and the kubelet’s KUBE-FIREWALL.
(nf_tables) means: iptables is just a frontend
for nftables here – nft list ruleset shows the same rules,
with the note “managed by iptables-nft, do not touch!”.
Tip: If a task asks “How are Services implemented?”,
iptables-save | grep -c KUBE-SVCandk -n kube-system get dsare the fastest clues.
Save your progress? Sign up for free
Drill 6: tcpdump on the node
The most important switches: -i any all interfaces,
-nn no host or port names (faster, unambiguous),
-c N stop after N packets, followed by a filter
(host, port, tcp,
udp, and, or, not).
The web Pod runs on worker2 – listen there:
ssh [email protected]
sudo tcpdump -i any -nn -c 4 host 10.0.2.248 and tcp port 80In a second terminal on cp1:
k exec client -- curl -s -o /dev/null -w '%{http_code}\n' http://10.0.2.248Expected (abridged, on worker2):
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
23:18:21.547215 cilium_vxlan P IP 10.0.1.27.40446 > 10.0.2.248.80: Flags [S], ...
23:18:21.547368 lxc451ed71841b8 Out IP 10.0.1.27.40446 > 10.0.2.248.80: Flags [S], ...
23:18:21.547412 lxc451ed71841b8 In IP 10.0.2.248.80 > 10.0.1.27.40446: Flags [S.], ...
23:18:21.547427 cilium_vxlan Out IP 10.0.2.248.80 > 10.0.1.27.40446: Flags [S.], ...
4 packets captured
With -i any, each packet is prefixed with its interface:
the SYN comes out of the tunnel already unwrapped
(cilium_vxlan) and goes through the lxc
interface into the Pod. From the host’s point of view, “into the Pod” is
Out. Use exit to get back to cp1.
Day 4 also looks at the packet on eth0, still
encapsulated.
Save your progress? Sign up for free
Drill 7: nsenter and k debug – into a Pod’s network
The nginx image has neither ip nor ss.
nsenter -n starts a host tool in the Pod’s
network namespace. You know the way there from the CKA: Pod → container
ID → PID. On worker2:
ssh [email protected]
sudo crictl pods --namespace tag01 --name web
ID=$(sudo crictl ps --name web -q --label io.kubernetes.pod.namespace=tag01)
P=$(sudo crictl inspect -o go-template --template '{{.info.pid}}' $ID)
echo $P
sudo nsenter -t $P -n ip -br addr
sudo nsenter -t $P -n ip route
sudo nsenter -t $P -n ss -tlnpExpected (abridged):
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
820ca44c235f8 About a minute ago Ready web tag01 0 (default)
25308
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0@if33 UP 10.0.2.248/32 fe80::5018:bdff:fe0c:74eb/64
default via 10.0.2.241 dev eth0 mtu 1450
10.0.2.241 dev eth0 scope link
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",...))
LISTEN 0 511 [::]:80 [::]:* users:(("nginx",...))
The Pod has a /32 address and a default route to
10.0.2.241 – that is cilium_host on worker2.
eth0@if33 reveals the peer on the host: interface number
33.
ip -o link | awk -F': ' '$1==33{print $2}'
sudo nsenter -t $P -n tcpdump -i eth0 -nn -c 2 icmp
exitExpected (abridged, after
k exec client -- ping -c 2 10.0.2.248 from a second
terminal on cp1):
lxc451ed71841b8@if32
23:18:51.458515 IP 10.0.1.27 > 10.0.2.248: ICMP echo request, id 30, seq 1, length 64
23:18:51.458630 IP 10.0.2.248 > 10.0.1.27: ICMP echo reply, id 30, seq 1, length 64
tcpdump in the Pod network sees only the traffic of this
one Pod – ideal when the node is busy.
A second way, without SSH to the node: k debug attaches
an ephemeral container to the running Pod. It shares
the Pod’s network, --target additionally its processes, and
--profile=netadmin gives it NET_ADMIN and
NET_RAW. Back on cp1:
k debug web -it --image=nicolaka/netshoot:v0.16 --target=web --profile=netadminIn the debug container’s shell:
ss -tlnp
tcpdump -i eth0 -nn -c 2 icmp
exitExpected (abridged, ping as above from
a second terminal):
Targeting container "web". If you don't see processes from this container it may be
because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-tnspq.
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1,...
23:18:51.458515 IP 10.0.1.27 > 10.0.2.248: ICMP echo request, id 30, seq 1, length 64
23:18:51.458630 IP 10.0.2.248 > 10.0.1.27: ICMP echo reply, id 30, seq 1, length 64
An ephemeral container cannot be removed – it stays
Terminated in the Pod until the Pod is deleted. The same
works for a node: k debug node/worker2 starts a Pod in the
node’s host network (host file system under /host):
k debug node/worker2 -it --image=nicolaka/netshoot:v0.16 --profile=netadmin
ip -br addr show cilium_host
exit
k get pods
k delete pod node-debugger-worker2-tpfh2 $nowExpected (abridged):
Creating debugging pod node-debugger-worker2-tpfh2 with container debugger on node
worker2.
cilium_host@cilium_net UP 10.0.2.241/32 fe80::501c:faff:fe6a:b5db/64
NAME READY STATUS RESTARTS AGE
client 1/1 Running 0 2m23s
node-debugger-worker2-tpfh2 0/1 Completed 0 5s
web 1/1 Running 0 2m24s
The node debug Pod stays behind as Completed after
exit – always clean up (take the name from
k get pods).
Save your progress? Sign up for free
Drill 8: cilium-dbg in the right agent
cilium-dbg is the debug CLI inside the agent container.
It only knows its own node. So always pick the agent of
the affected node first:
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --brief
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
| grep -E 'ENDPOINT|10.0.2.248'
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg identity get 5472
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg node list
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache get 10.0.1.27Expected (abridged):
OK
ENDPOINT POLICY (ingress) ... IDENTITY LABELS IPv4 STATUS
755 Disabled ... 5472 k8s:app=web 10.0.2.248 ready
ID LABELS
5472 k8s:app=web
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=tag01
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=tag01
Name IPv4 Address Endpoint CIDR IPv6 Address Endpoint CIDR Source
cp1 172.16.0.126 10.0.0.0/24 custom-resource
worker1 172.16.0.228 10.0.1.0/24 custom-resource
worker2 172.16.0.83 10.0.2.0/24 local
10.0.1.27 maps to identity identity=11890 encryptkey=0 tunnelendpoint=172.16.0.228
flags=hastunnel
Take the identity number from endpoint list. The ipcache
is Cilium’s address book: for every IP the identity and – for remote
Pods – the node the tunnel leads to. Other subcommands you need in this
book: service list, monitor --type drop,
bpf lb list, policy get,
config.
Save your progress? Sign up for free
Exercise: Speed Drills
- Which process listens on port 4244 on
cp1? Target time: 15 s - Through which interface does a packet from
cp1to10.0.1.27leave the host? Target time: 10 s - Show that there are no kube-proxy Service chains in
iptables. Target time: 10 s - Show the endpoint ID and identity of the Pod
clientin the responsible Cilium agent. Target time: 40 s
Solution
sudo ss -tlnp '( sport = :4244 )'
ip route get 10.0.1.27
sudo iptables-save | grep -c KUBE-SVC
N=$(k get pod client -o jsonpath='{.spec.nodeName}')
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=$N -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
| grep -E 'ENDPOINT|10.0.1.27'Expected (abridged):
LISTEN 0 4096 *:4244 *:* users:(("cilium-agent",...))
10.0.1.27 dev cilium_host src 10.0.0.88 uid 1000
cache mtu 1450
0
ENDPOINT POLICY (ingress) ... IDENTITY LABELS IPv4 STATUS
1384 Disabled ... 11890 k8s:io... 10.0.1.27 ready
The endpoint line shows only the first label;
cilium-dbg identity get shows all of them.
Save your progress? Sign up for free
Checkpoint
type kExpected: k is aliased to `kubectl'
k config current-contextExpected: tag01
k get pods -o wideExpected: client and web
Running, no node-debugger-… Pod left.
k get pod web -o jsonpath='{.spec.ephemeralContainers[*].name}'; echoExpected: a name like debugger-tnspq –
the ephemeral container from Drill 7.
sudo iptables-save | grep -c KUBE-SVCExpected: 0
Challenge
Find out what the lxc interface of the Pod
web is called on its node – using only k and
cilium-dbg, without SSH – and then capture a
curl on the node on exactly that interface.
Hint:
cilium-dbg endpoint get <id> -o jsonpath=… returns a
list; the field is called
status.networking.interface-name.
Solution
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint get 755 \
-o jsonpath='{[0].status.networking.interface-name}{"\n"}'
ssh [email protected] sudo tcpdump -i lxc451ed71841b8 -nn -c 2 tcp port 80In the second terminal:
k exec client -- curl -s -o /dev/null http://10.0.2.248.
Expected (abridged):
lxc451ed71841b8
... IP 10.0.1.27.40446 > 10.0.2.248.80: Flags [S], ...
... IP 10.0.2.248.80 > 10.0.1.27.40446: Flags [S.], ...
Save your progress? Sign up for free
Cleanup
k config use-context kubernetes-admin@kubernetes
k config delete-context tag01
k delete ns tag01Nothing changed cluster-wide today. The image
nicolaka/netshoot:v0.16 is now cached on the nodes – Day 4
uses it again.
Exam Tip
- Every task starts with the given context or
sshcommand;exitat the end, otherwise you work on the wrong host in the next task. - Always run
cilium-dbgin the agent of the affected node (--field-selector spec.nodeName=…);ds/ciliumhits a random agent. - Docs: kubernetes.io “Debug Running Pods”
(
kubectl debug, ephemeral containers) and docs.cilium.io “Troubleshooting” and “Command Cheatsheet”.
Enterprise Note
kubectl debugandnodes/proxyare powerful: allow ephemeral containers and node debug Pods only for operations teams, via RBAC and Pod Security Admission.- Captures on production nodes often contain plaintext (HTTP, DNS):
only with a filter, a limited duration (
-c,-G) and after approval. - Pull debug images like netshoot from your own registry, with a fixed
version and a signature – not
latestfrom Docker Hub. - Instead of
tcpdumpon every node: Hubble (Day 20) shows flows cluster-wide with Pod names and policy verdicts.