Day 21: Competition – Repairing the Broken Network
Goal: You find and fix eight planted network faults across the CKNE curriculum – against the clock, with proof before and after. · Duration: approx. 90 min · Prerequisite: Days 0–20
Briefing
- All five curriculum domains: Core Infrastructure and CNI, Service Networking and DNS, Advanced Traffic Management, Network Security and Policy, Observability.
- Eight rounds, each in its own namespace (
tag21-r1…tag21-r8). Round 1 (agent), round 3 (CoreDNS) and round 7 (BGP) plant the fault in the cluster itself – you roll it back. - Tools as in the exam:
k get/describe, status conditions,cilium,cilium-dbg,hubble observe,curl,dig,sshto nodes androuter. - Every round ends with a proof command: broken before, fixed after.
- New in this Competition:
CiliumNodeConfig(Cilium values per node, they override the ConfigMap;k get ciliumnodeconfigs -A) andhubble observe --drop-reason-desc(filter drops by reason). Both are also in the cheat sheet.
Round Fault Tool
1 Pods on worker2 isolated cilium-dbg status, CiliumNodeConfig
2 Service without backends EndpointSlice, port names
3 wrong answer for a Service name dig, Corefile
4 Default Deny eats DNS hubble observe, NetworkPolicy
5 app reports 403 hubble --protocol http, CNP L7
6 HTTPS at the Gateway broken curl --cacert, Route status
7 LoadBalancer IP dead from outside cilium bgp, FRR
8 failover to west does not kick in clustermesh, global annotation
mkdir -p ~/ckne/tag21 && cd ~/ckne/tag21Rules of Play
- Per round: run the setup commands of the game master, then start the clock. Do not study the manifest – in the exam, too, you only see it once you go looking.
- Find the cause, fix it, prove it with one command. Only then stop the clock.
- Fix the cause, not the symptom: whoever deletes a policy instead of correcting it, or moves Pods to a different node, gets zero points.
- Only read the Hint once the target time has run out.
Every round except round 8 starts the same way (round 8 creates the namespaces in both clusters itself), here for round 1:
k create ns tag21-r1
k config set-context --current --namespace=tag21-r1Round 1: Island worker2
Situation: After a maintenance window, teams report:
Pods on worker2 no longer resolve names and cannot reach
Pods on other nodes. All Cilium Pods are Running. Find the
cause and restore the normal state.
Game master:
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
--replicas=2 --port=8080 -- /agnhost netexec --http-port=8080
k expose deployment echo --port=80 --target-port=8080
k run c1 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k run c2 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k wait --for=condition=Ready pod --all
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNodeConfig
metadata:
name: worker2-tuning
namespace: kube-system
spec:
nodeSelector:
matchLabels:
kubernetes.io/hostname: worker2
defaults:
tunnel-port: "8473"
EOF
k -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=worker2
k -n kube-system rollout status ds/ciliumTarget time: 6 min
Hint: Configuration does not only come from the
ConfigMap cilium-config. The agent’s init container log
reveals which sources it has read.
Solution
Symptoms and a first look at the agents:
k exec c2 -- curl -sS -m3 echo/hostname; echo " rc=$?"
cilium status | grep -E 'Cilium:|Hubble Relay:'
for n in worker1 worker2; do
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=$n -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status \
| grep 'Cluster health'
doneExpected (abridged):
curl: (28) Resolving timed out after 3002 milliseconds
command terminated with exit code 28
rc=28
/¯¯\__/¯¯\ Cilium: 1 errors
\__/¯¯\__/ Hubble Relay: 2 errors
Cluster health: 3/4 reachable (2026-11-19T02:29:20Z) ...
Cluster health: 1/3 reachable (2026-11-19T02:27:57Z) ...
cilium-health checks every node through the tunnel:
worker2 only reaches itself, the others do not reach
worker2 (the 4 includes west/argo). Where does
the agent on worker2 get its configuration from
($CP points to it after the loop)?
k -n kube-system logs $CP -c config | grep 'Got configuration' | grep -o 'lenC.*'
k get ciliumnodeconfigs -A
k -n kube-system get ciliumnodeconfig worker2-tuning -o jsonpath='{.spec.defaults}'
echo
ssh [email protected] 'ip -d link show cilium_vxlan 2>/dev/null' \
| grep -o 'dstport [0-9]*'
ssh [email protected] 'ip -d link show cilium_vxlan 2>/dev/null' \
| grep -o 'dstport [0-9]*'Expected (abridged):
lenConfigPairs=188 configSource="{Kind:config-map Namespace:kube-system Name:cil...
lenConfigPairs=1 configSource="{Kind:cilium-node-config Namespace:kube-system ...
NAMESPACE NAME AGE
kube-system worker2-tuning 50m
{"tunnel-port":"8473"}
dstport 8472
dstport 8473
A CiliumNodeConfig overrides values of the ConfigMap for
selected nodes. Here worker2 encapsulates VXLAN on UDP
8473, all others on 8472 – the packets arrive, but nobody unpacks them.
cilium config view does not show this, it only reads the
ConfigMap. Repair: delete the override, restart the agent:
k -n kube-system delete ciliumnodeconfig worker2-tuning
k -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=worker2
k -n kube-system rollout status ds/cilium
ssh [email protected] 'ip -d link show cilium_vxlan 2>/dev/null' \
| grep -o 'dstport [0-9]*'Expected:
ciliumnodeconfig.cilium.io "worker2-tuning" deleted from kube-system namespace
pod "cilium-nwzn2" deleted from kube-system namespace
daemon set "cilium" successfully rolled out
dstport 8472
After one to two minutes (until the other agents have registered the node again):
k exec c2 -- curl -s -m3 echo/hostname; echo
for i in $(seq 8); do k exec c1 -- curl -s -m2 echo/hostname; echo; done \
| sort | uniq -cExpected:
echo-cc8555f69-x8r52
2 echo-cc8555f69-574ns
6 echo-cc8555f69-x8r52
Both backends answer, across the node boundary too. Whoever only restarts the Pods instead changes nothing: the fault sits in the node’s data path.
Save your progress? Sign up for free
Round 2: The Service without a target
Situation: In tag21-r2, two healthy
notes-api Pods are running, readiness green. The Service
notes still refuses every connection. Repair the Service
without touching the Deployment.
Game master (namespace and context as above, here
tag21-r2):
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 2
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: notes-api
image: ghcr.io/thomaszachmann/notes-api:0.5.1
env:
- name: NOTES_API_KEY
value: lab-key-123
ports:
- name: api
containerPort: 8080
readinessProbe:
httpGet:
path: /healthz
port: api
---
apiVersion: v1
kind: Service
metadata:
name: notes
spec:
selector:
app: notes
ports:
- name: http
port: 80
targetPort: http
EOF
k run c --image=curlimages/curl:8.17.0 --command -- sleep infinity
k rollout status deploy notesTarget time: 3 min
Hint: The EndpointSlice has addresses. Look at the column next to them.
Solution
k exec c -- curl -sS -m3 notes/version; echo " rc=$?"
k get endpointslices
k describe svc notes | grep -E 'TargetPort|Endpoints'
k get pods -l app=notes -o jsonpath='{.items[0].spec.containers[0].ports}'; echo
CIP=$(k get svc notes -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A2 $CIPExpected:
curl: (7) Failed to connect to notes port 80 after 3 ms: Could not connect to server
command terminated with exit code 7
rc=7
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
notes-9mbxg IPv4 <unset> 10.0.1.2,10.0.2.71 36m
TargetPort: http/TCP
Endpoints: 10.0.1.2,10.0.2.71
[{"containerPort":8080,"name":"api","protocol":"TCP"}]
75 10.96.118.49:80/TCP ClusterIP
A named targetPort is resolved per Pod via the name of
the container port. No Pod has a port http – the slice
knows the addresses, but no port. Cilium therefore programs the Service
without a backend and rejects every connection immediately (exit 7, no
Hubble flow).
k patch svc notes -p '{"spec":{"ports":[{"port":80,"targetPort":"api"}]}}'
k get endpointslices
k exec c -- curl -s -m3 notes/version; echo
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A2 $CIPExpected:
service/notes patched
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
notes-9mbxg IPv4 8080 10.0.2.71,10.0.1.2 36m
0.5.1
75 10.96.118.49:80/TCP ClusterIP 1 => 10.0.1.2:8080/TCP (active)
2 => 10.0.2.71:8080/TCP (active)
The patch hits the port entry via its key port: 80; the
name http is kept.
Save your progress? Sign up for free
Round 3: The wrong twin
Situation: In tag21-r3 there are
api (new) and api-v1 (old). Clients that call
api end up at the old service. Deployments and Services are
correct.
Game master (backup first – it serves as proof at the end):
for d in api api-v1; do
k create deployment $d --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
--port=8080 -- /agnhost netexec --http-port=8080
k expose deployment $d --port=80 --target-port=8080
done
k run c --image=nicolaka/netshoot:v0.16 -- sleep infinity
k -n kube-system get cm coredns -o yaml > r3-coredns-backup.yaml
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > r3-Corefile
R='rewrite name exact api.tag21-r3.svc.cluster.local'
sed -i "s/^ ready$/ ready\n $R api-v1.tag21-r3.svc.cluster.local/" r3-Corefile
k -n kube-system create configmap coredns --from-file=Corefile=r3-Corefile $do \
| k replace -f -Wait about a minute until both CoreDNS Pods have reloaded.
Target time: 5 min
Hint: Compare the IP from DNS with
k get svc. Whoever answers names in the cluster reads its
rules from a ConfigMap.
Solution
k exec c -- curl -s -m3 api/hostname; echo
k get svc
k exec c -- dig api.tag21-r3.svc.cluster.local | grep -A1 'ANSWER SECTION'
k exec c -- curl -s -m3 $(k get svc api -o jsonpath='{.spec.clusterIP}')/hostname
echoExpected:
api-v1-9cc68fb5b-8n6xb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api ClusterIP 10.97.130.83 <none> 80/TCP 89s
api-v1 ClusterIP 10.99.19.37 <none> 80/TCP 88s
;; ANSWER SECTION:
api.tag21-r3.svc.cluster.local. 30 IN A 10.99.19.37
api-98c78c9bb-nt8wx
By IP the right Pod answers, by name the wrong one: the fault lies in
DNS, not in the Service. The answer carries the requested name, but the
IP of api-v1 – typical of rewrite, which
translates the name in the answer back. Check and correct the
Corefile:
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -n rewrite
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
sed -i '/rewrite name exact api.tag21-r3/d' Corefile
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -Expected:
7: rewrite name exact api.tag21-r3.svc.cluster.local api-v1.tag21-r3.svc.cluster...
configmap/coredns replaced
After about a minute (each CoreDNS Pod reloads on its own, Day 8):
k -n kube-system logs -l k8s-app=kube-dns --since=2m --prefix | grep 'Reloading c'
k exec c -- dig +short api.tag21-r3.svc.cluster.local
k exec c -- curl -s -m3 api/hostname; echo
diff <(k -n kube-system get cm coredns -o jsonpath='{.data}') \
<(k create -f r3-coredns-backup.yaml $do -o jsonpath='{.data}') && echo IDENTISCHExpected (abridged):
[pod/coredns-7cd4959b5-87shd/coredns] [INFO] Reloading complete
[pod/coredns-7cd4959b5-nkb2d/coredns] [INFO] Reloading complete
10.97.130.83
api-98c78c9bb-nt8wx
IDENTISCH
Only when both Pods report “Reloading complete” is the repair done – until then, every second lookup still answers wrongly.
Save your progress? Sign up for free
Round 4: Silence after Default Deny
Situation: In tag21-r4, Default Deny
applies in both directions. web is supposed to reach
notes via the Service name, no other Pod. The policies are
“done”, yet web only gets timeouts.
Game master:
k create deployment notes --image=ghcr.io/thomaszachmann/notes-api:0.5.1 \
--port=8080 $do > r4-notes.yaml
k set env --local -f r4-notes.yaml NOTES_API_KEY=lab-key-123 -o yaml | k apply -f -
k expose deployment notes --port=80 --target-port=8080
k run web --image=curlimages/curl:8.17.0 -l app=web --command -- sleep infinity
k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-to-notes
spec:
podSelector:
matchLabels:
app: web
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: notes
ports:
- port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: notes-from-web
spec:
podSelector:
matchLabels:
app: notes
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: web
ports:
- port: 8080
EOF
k rollout status deploy notesTarget time: 4 min
Hint: Test once with the name, once with the ClusterIP. Hubble tells you the destination of the dropped packet.
Solution
k exec web -- curl -sS -m5 notes/version; echo " rc=$?"
k exec web -- curl -sS -m5 $(k get svc notes -o jsonpath='{.spec.clusterIP}')/version
echo
hubble observe -P --port-forward-port 0 --from-pod tag21-r4/web --verdict DROPPED \
--last 2Expected (abridged):
curl: (28) Resolving timed out after 5000 milliseconds
command terminated with exit code 28
rc=28
0.5.1
tag21-r4/web:32987 (ID:95867) <> kube-system/coredns-7cd4959b5-nkb2d:53 (ID:70078)
policy-verdict:none EGRESS DENIED (UDP)
tag21-r4/web:32987 (ID:95867) <> kube-system/coredns-7cd4959b5-nkb2d:53 (ID:70078)
Policy denied DROPPED (UDP)
By IP it works – the rules for 8080 are right. What gets dropped is
DNS. Which label does allow-dns expect, which one does the
namespace have?
k describe netpol allow-dns | grep -A2 'To:'
k get ns kube-system --show-labelsExpected:
To:
NamespaceSelector: name=kube-system
PodSelector: k8s-app=kube-dns
NAME STATUS AGE LABELS
kube-system Active 4h kubernetes.io/metadata.name=kube-system
Nobody sets the label name; the only reliable one is
kubernetes.io/metadata.name, which the API server gives
every namespace:
k patch netpol allow-dns --type=json -p '[{"op":"replace",
"path":"/spec/egress/0/to/0/namespaceSelector/matchLabels",
"value":{"kubernetes.io/metadata.name":"kube-system"}}]'
k exec web -- curl -sS -m5 notes/version; echo
k run fremd --image=curlimages/curl:8.17.0 --rm -i --restart=Never --command -- \
curl -sS -m3 notes/version; echo " rc=$?"Expected (abridged):
networkpolicy.networking.k8s.io/allow-dns patched
0.5.1
curl: (28) Connection timed out after 3002 milliseconds
pod "fremd" deleted from tag21-r4 namespace
pod tag21-r4/fremd terminated (Error)
rc=28
fremd now resolves the name, but does not get past
notes – Default Deny still applies.
Save your progress? Sign up for free
Round 5: The order with a 403
Situation: The Pod shop in
tag21-r5 calls four notes-api endpoints in a loop and
reports errors in its log without saying which call fails. Allow exactly
the missing call – nothing more.
Game master:
k create deployment notes --image=ghcr.io/thomaszachmann/notes-api:0.5.1 \
--port=8080 $do > r5-notes.yaml
k set env --local -f r5-notes.yaml NOTES_API_KEY=lab-key-123 -o yaml | k apply -f -
k expose deployment notes --port=80 --target-port=8080
k run shop --image=curlimages/curl:8.17.0 -l app=shop --command -- sh -c '
while true; do
for r in "GET /version" "GET /notes" "POST /notes" "GET /healthz"; do
set -- $r
c=$(curl -s -o /dev/null -w "%{http_code}" -m3 -X $1 \
-H "X-API-Key: lab-key-123" -d "{\"text\":\"bestellung\"}" notes$2)
[ "$c" -ge 400 ] && echo "$(date +%T) FEHLER $c"
done
sleep 5
done'
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: notes-l7
spec:
endpointSelector:
matchLabels:
app: notes
ingress:
- fromEndpoints:
- matchLabels:
app: shop
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: GET
path: "/(version|healthz|notes)"
EOF
k rollout status deploy notesTarget time: 4 min
Hint: A 403 without a timeout comes from Envoy. Hubble shows dropped requests with method and URL.
Solution
k logs shop --tail=2
hubble observe -P --port-forward-port 0 -n tag21-r5 --verdict DROPPED \
--protocol http --last 2Expected (abridged):
02:38:08 FEHLER 403
02:38:13 FEHLER 403
tag21-r5/shop:37468 (ID:78769) -> tag21-r5/notes-5f985975d8-p4k6r:8080 (ID:75232)
http-request DROPPED (HTTP/1.1 POST http://notes/notes)
tag21-r5/shop:37498 (ID:78769) -> tag21-r5/notes-5f985975d8-p4k6r:8080 (ID:75232)
http-request DROPPED (HTTP/1.1 POST http://notes/notes)
Only POST /notes is missing. Append a rule instead of
loosening the existing one (method: ".*" would also open
DELETE):
k patch cnp notes-l7 --type=json -p '[{"op":"add",
"path":"/spec/ingress/0/toPorts/0/rules/http/-",
"value":{"method":"POST","path":"/notes"}}]'
k get cnp notes-l7 -o jsonpath='{.spec.ingress[0].toPorts[0].rules.http}'; echoExpected:
ciliumnetworkpolicy.cilium.io/notes-l7 patched
[{"method":"GET","path":"/(version|healthz|notes)"},{"method":"POST","path":"/notes"}]
After 15 seconds:
k logs shop --since=12s | wc -l
k exec shop -- curl -s -m3 -X DELETE notes/notes; echo
hubble observe -P --port-forward-port 0 --to-pod tag21-r5/notes --protocol http \
--http-method POST --last 1Expected (abridged):
0
Access denied
tag21-r5/shop:46614 (ID:78769) -> tag21-r5/notes-5f985975d8-p4k6r:8080 (ID:75232)
http-request FORWARDED (HTTP/1.1 POST http://notes/notes)
No more errors in the log, POST works,
DELETE stays forbidden.
Save your progress? Sign up for free
Round 6: The payment Gateway
Situation: The platform team runs the Gateway
edge in tag21-r6-infra with a certificate from
its own CA. The app team has created the route pay in
tag21-r6.
https://pay.example.com:<NodePort>/ is supposed to
answer cleanly with --cacert. The route stays in the app
namespace.
Game master:
k create ns tag21-r6-infra
k apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned
namespace: tag21-r6-infra
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: lab-ca
namespace: tag21-r6-infra
spec:
isCA: true
commonName: Tag21 CA
secretName: lab-ca
privateKey:
algorithm: ECDSA
issuerRef:
name: selfsigned
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: lab-ca
namespace: tag21-r6-infra
spec:
ca:
secretName: lab-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: pay
namespace: tag21-r6-infra
spec:
secretName: pay-tls
dnsNames:
- shop.example.com
issuerRef:
name: lab-ca
---
apiVersion: cilium.io/v2alpha1
kind: CiliumGatewayClassConfig
metadata:
name: nodeport
namespace: tag21-r6-infra
spec:
service:
type: NodePort
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: tag21-nodeport
spec:
controllerName: io.cilium/gateway-controller
parametersRef:
group: cilium.io
kind: CiliumGatewayClassConfig
name: nodeport
namespace: tag21-r6-infra
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: edge
namespace: tag21-r6-infra
spec:
gatewayClassName: tag21-nodeport
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: pay.example.com
tls:
mode: Terminate
certificateRefs:
- name: pay-tls
allowedRoutes:
namespaces:
from: Same
EOF
k create deployment pay --image=registry.k8s.io/gateway-api/echo-basic:v1.5.1
k expose deployment pay --port=8080 --target-port=3000
k apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: pay
namespace: tag21-r6
spec:
parentRefs:
- name: edge
namespace: tag21-r6-infra
sectionName: https
hostnames:
- pay.example.com
rules:
- backendRefs:
- name: pay
port: 8080
EOF
k -n tag21-r6-infra wait --for=condition=Programmed gateway edgeFor testing (part of the task, not of the fault):
NPS=$(k -n tag21-r6-infra get svc cilium-gateway-edge \
-o jsonpath='{.spec.ports[0].nodePort}')
k -n tag21-r6-infra get secret lab-ca -o jsonpath='{.data.ca\.crt}' | base64 -d \
> r6-ca.crt
R="--resolve pay.example.com:$NPS:172.16.0.228"Target time: 6 min
Hint: Two faults in a row. The first is in the certificate itself, the second in the status of the route.
Solution
k -n tag21-r6-infra get gateway edge
curl -sS $R --cacert r6-ca.crt https://pay.example.com:$NPS/; echo "exit=$?"
k -n tag21-r6-infra get secret pay-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -ext subjectAltNameExpected (abridged):
NAME CLASS ADDRESS PROGRAMMED AGE
edge tag21-nodeport 172.16.0.83 True 3s
curl: (60) SSL: no alternative certificate subject name matches target host name
'pay.example.com'
exit=60
X509v3 Subject Alternative Name: critical
DNS:shop.example.com
Gateway Programmed, listener ResolvedRefs –
as far as Kubernetes is concerned, everything is valid. Only the name in
the certificate does not match the listener. The certificate belongs to
cert-manager, so correct it there, not in the Secret:
k -n tag21-r6-infra patch certificate pay --type=merge \
-p '{"spec":{"dnsNames":["pay.example.com"]}}'Wait a few seconds until cert-manager has reissued it, then check:
k -n tag21-r6-infra get secret pay-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -ext subjectAltName
curl -s $R --cacert r6-ca.crt https://pay.example.com:$NPS/ -w '%{http_code}\n'
RC='{range .status.parents[*].conditions[*]}{.type}={.status} '\
'{.reason}: {.message}{"\n"}{end}'
k get httproute pay -o jsonpath="$RC"Expected:
certificate.cert-manager.io/pay patched
X509v3 Subject Alternative Name: critical
DNS:pay.example.com
404
Accepted=False NotAllowedByListeners: HTTPRoute is not allowed to attach to this
Gateway due to namespace restrictions
ResolvedRefs=True ResolvedRefs: Service reference is valid
TLS works, but Envoy has no route for the host (404). The listener
only accepts routes from its own namespace. Allow exactly the app
namespace, not All:
k -n tag21-r6-infra patch gateway edge --type=json -p '[{"op":"replace",
"path":"/spec/listeners/0/allowedRoutes/namespaces",
"value":{"from":"Selector","selector":{"matchLabels":
{"kubernetes.io/metadata.name":"tag21-r6"}}}}]'
k get httproute pay -o jsonpath="$RC" | cut -d: -f1
k -n tag21-r6-infra get gateway edge -o jsonpath='{.status.listeners[0].attachedRoutes}'
echo
curl -s $R --cacert r6-ca.crt https://pay.example.com:$NPS/ | grep '"host"'Expected:
gateway.gateway.networking.k8s.io/edge patched
Accepted=True Accepted
ResolvedRefs=True ResolvedRefs
1
"host": "pay.example.com:32668",
A ReferenceGrant would not have helped here: it governs references
from the route to foreign objects; attaching
to a foreign Gateway is governed by
allowedRoutes.
Save your progress? Sign up for free
Round 7: The invisible Service IP
Situation: The LoadBalancer Service
echo in tag21-r7 has an IP from
10.50.0.0/24. It is supposed to be reachable from
router via BGP, and according to the network team the
sessions are up. Still: timeout.
Game master:
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
--port=8080 -- /agnhost netexec --http-port=8080
k expose deployment echo --type=LoadBalancer --port=80 --target-port=8080 \
--labels=bgp=on
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
name: tag21-pool
spec:
allowFirstLastIPs: "No"
blocks:
- cidr: 10.50.0.0/24
serviceSelector:
matchLabels:
io.kubernetes.service.namespace: tag21-r7
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
name: tag21-bgp
spec:
nodeSelector:
matchLabels:
kubernetes.io/os: linux
bgpInstances:
- name: ost
localASN: 65010
peers:
- name: router
peerASN: 65000
peerAddress: 172.16.0.123
peerConfigRef:
name: tag21-peer
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
name: tag21-peer
spec:
families:
- afi: ipv4
safi: unicast
advertisements:
matchLabels:
advertise: tag21
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
name: tag21-lb
labels:
advertise: bgp
spec:
advertisements:
- advertisementType: Service
service:
addresses:
- LoadBalancerIP
selector:
matchLabels:
bgp: "on"
EOF
k rollout status deploy echoTarget time: 5 min
Hint: A session being “established” does not mean “announces something”. Follow the chain ClusterConfig → PeerConfig → Advertisement via labels.
Solution
k get svc echo
ssh [email protected] curl -sS -m3 http://10.50.0.1/hostname; echo " rc=$?"
ssh [email protected] ip route get 10.50.0.1
cilium bgp peers
cilium bgp routes advertised ipv4 unicastExpected (abridged):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo LoadBalancer 10.99.60.2 10.50.0.1 80:30378/TCP 21s
curl: (28) Connection timed out after 3002 milliseconds
rc=28
10.50.0.1 via 172.16.0.1 dev eth0 src 172.16.0.123 uid 1000
Node Local AS Peer AS Peer Address Session State ... Received Advertised
cp1 65010 65000 172.16.0.123 established ... 0 0
worker1 65010 65000 172.16.0.123 established ... 0 0
worker2 65010 65000 172.16.0.123 established ... 0 0
Node VRouter Peer Prefix NextHop Age Attrs
IP assigned, sessions up, but nothing announced – router
sends 10.50.0.1 to its default gateway. What is the PeerConfig looking
for, and what is the Advertisement labeled?
k get ciliumbgppeerconfig tag21-peer \
-o jsonpath='{.spec.families[0].advertisements}'; echo
k get ciliumbgpadvertisements --show-labelsExpected:
{"matchLabels":{"advertise":"tag21"}}
NAME AGE LABELS
tag21-lb 29s advertise=bgp
The PeerConfig’s selector finds no Advertisement. Set the label to match:
k label ciliumbgpadvertisement tag21-lb advertise=tag21 --overwrite
cilium bgp routes advertised ipv4 unicast
ssh [email protected] ip route show 10.50.0.1
ssh [email protected] curl -s -m3 http://10.50.0.1/hostname; echoExpected (abridged):
ciliumbgpadvertisement.cilium.io/tag21-lb labeled
Node VRouter Peer Prefix NextHop Age Attrs
cp1 65010 172.16.0.123 10.50.0.1/32 172.16.0.126 9s [{Origin: i} ...
worker1 65010 172.16.0.123 10.50.0.1/32 172.16.0.228 9s [{Origin: i} ...
worker2 65010 172.16.0.123 10.50.0.1/32 172.16.0.83 9s [{Origin: i} ...
10.50.0.1 nhid 85 proto bgp metric 20
nexthop via 172.16.0.83 dev eth0 weight 1
nexthop via 172.16.0.126 dev eth0 weight 1
nexthop via 172.16.0.228 dev eth0 weight 1
echo-cc8555f69-4p8ld
Three equal-cost paths via ECMP, router reaches the
Service.
Save your progress? Sign up for free
Round 8: Failover into the void
Situation: inventory runs in
ost and west, namespace tag21-r8.
According to the operations manual, west takes over when
ost has no backends – which is why ost is
scaled to 0 for maintenance. The client in ost still gets
errors. Without scaling ost back up: restore the
failover.
Game master (namespace in both clusters):
for c in ost west; do
k --context $c create ns tag21-r8
k --context $c -n tag21-r8 create deployment inventory-$c \
--image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --port=8080 \
-- /agnhost netexec --http-port=8080
k --context $c -n tag21-r8 expose deployment inventory-$c --name=inventory \
--port=80 --target-port=8080
done
k config set-context --current --namespace=tag21-r8
k run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k annotate svc inventory service.cilium.io/global=true \
service.cilium.io/affinity=local
k --context west -n tag21-r8 rollout status deploy inventory-west
k wait --for=condition=Ready pod/client
k scale deployment inventory-ost --replicas=0Target time: 4 min
Hint: A global Service has two halves. Check both clusters.
Solution
k exec client -- curl -sS -m3 inventory/hostname; echo " rc=$?"
cilium clustermesh status | grep -E 'west:|nodes are connected'
k get svc inventory -o jsonpath='{.metadata.annotations}'; echo
k --context west -n tag21-r8 get svc inventory -o jsonpath='{.metadata.annotations}'
echo
CIP=$(k get svc inventory -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
--clustermesh-affinity | grep -A1 $CIPExpected (abridged, without the CLI’s symbols):
curl: (7) Failed to connect to inventory:80 after 29 ms: Could not connect to server
command terminated with exit code 7
rc=7
All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
- west: 3/3 configured, 3/3 connected - KVStoreMesh: 1/1 configured, 1/1 connected
{"service.cilium.io/affinity":"local","service.cilium.io/global":"true"}
88 10.103.255.118:80/TCP ClusterIP
The mesh is up, ost is global – but the Service in
west has no annotation and does not share its backends. So
there is no backend at all, exit 7.
k --context west -n tag21-r8 annotate svc inventory service.cilium.io/global=true
k exec client -- curl -s -m3 inventory/hostname; echo
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
--clustermesh-affinity | grep -A1 $CIPExpected:
service/inventory annotated
inventory-west-66db64f486-dr5ss
88 10.103.255.118:80/TCP ClusterIP 1 => 10.1.0.201:8080/TCP (active) ...
The backend from 10.1.0.0/16 (west) is there. End of
maintenance – with affinity=local, traffic returns on its
own:
k scale deployment inventory-ost --replicas=1
k rollout status deployment inventory-ost
for i in 1 2 3 4; do k exec client -- curl -s -m3 inventory/hostname; echo; done \
| sort | uniq -cExpected:
deployment.apps/inventory-ost scaled
deployment "inventory-ost" successfully rolled out
4 inventory-ost-976947bf5-hn8hh
Stop the clock.
Save your progress? Sign up for free
Scoring
| Round | Fault | Target time | Points |
|---|---|---|---|
| 1 | CiliumNodeConfig with wrong tunnel port | 6 min | 3 |
| 2 | named targetPort without a match | 3 min | 2 |
| 3 | CoreDNS rewrite to old Service | 5 min | 2 |
| 4 | DNS exception with wrong namespace label | 4 min | 2 |
| 5 | L7 rule without POST /notes | 4 min | 2 |
| 6 | wrong SAN + allowedRoutes Same | 6 min | 3 |
| 7 | Advertisement label does not match the PeerConfig | 5 min | 2 |
| 8 | west Service not global | 4 min | 2 |
You only get full points within the target time, without a hint
and with a proof command; otherwise half (round down).
Whoever removes the symptom instead of the cause (deleting a policy,
moving Pods, allowedRoutes: All, method: ".*")
gets 0 for the round. Overall target time: 37 min, maximum: 18
points.
| Points | Assessment |
|---|---|
| 15–18 | exam-ready – move on to the Practice Exam (Appendix B) |
| 10–14 | good; repeat the days of the rounds you missed |
| 0–9 | work through Days 4, 8, 10 and 12 once more |
Challenge
Somewhere in the cluster, a policy drops traffic between two Pods. Find source, destination and policy using Hubble only, without knowing the namespace, and remove the leftover. Game master:
k create ns tag21-ch
k -n tag21-ch create deployment api \
--image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --port=8080 \
-- /agnhost netexec --http-port=8080
k -n tag21-ch expose deployment api --port=80 --target-port=8080
k -n tag21-ch run batch --image=curlimages/curl:8.17.0 -l app=batch --command -- \
sh -c 'while true; do curl -s -m2 -o /dev/null api/hostname; sleep 3; done'
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: tag21-ch-altlast
spec:
endpointSelector:
matchLabels:
app: api
enableDefaultDeny:
ingress: false
ingressDeny:
- fromEndpoints:
- matchLabels:
app: batch
EOFHint: hubble observe --verdict DROPPED
shows all drops – in the author’s lab mostly VLAN broadcasts from the
home network (Day 0). --drop-reason-desc filters by reason;
the policy reasons are called POLICY_DENIED and
POLICY_DENY.
Solution
hubble observe -P --port-forward-port 0 --verdict DROPPED --last 2
hubble observe -P --port-forward-port 0 --verdict DROPPED \
--drop-reason-desc POLICY_DENY --drop-reason-desc POLICY_DENIED --since 5m \
| tail -2Expected (abridged, depends on the network – without
VLAN broadcasts, the two VLAN_FILTERED lines are missing
for you, and the first command already shows the policy drops):
10.10.10.0:36526 (world) <> 10.10.11.255:10001 (world) VLAN traffic disallowed by
VLAN filter DROPPED (UDP)
192.168.3.1:36526 (world) <> 192.168.3.255:10001 (world) VLAN traffic disallowed by
VLAN filter DROPPED (UDP)
tag21-ch/batch:60336 (ID:71887) <> tag21-ch/api-98c78c9bb-vgt7r:8080 (ID:94484)
policy-verdict:L3-Only INGRESS DENIED (TCP Flags: SYN)
tag21-ch/batch:60336 (ID:71887) <> tag21-ch/api-98c78c9bb-vgt7r:8080 (ID:94484)
Policy denied by denylist DROPPED (TCP Flags: SYN)
“denied by denylist” means: an explicit deny rule, not Default Deny. There is none in the namespace – so search cluster-wide:
k -n tag21-ch get cnp,netpol
k get ccnp -o json | jq -r '.items[]
| select(.spec.ingressDeny or .spec.egressDeny) | .metadata.name'
k delete ccnp tag21-ch-altlast
hubble observe -P --port-forward-port 0 --from-pod tag21-ch/batch --to-port 8080 \
--last 1Expected (abridged):
No resources found in tag21-ch namespace.
tag21-ch-altlast
ciliumclusterwidenetworkpolicy.cilium.io "tag21-ch-altlast" deleted
tag21-ch/batch:56220 (ID:71887) -> tag21-ch/api-98c78c9bb-vgt7r:8080 (ID:94484)
to-endpoint FORWARDED (TCP Flags: ACK)
Save your progress? Sign up for free
Cleanup
The cluster changes from rounds 1 and 3 are already rolled back. What
goes are the BGP objects from round 7 (the BGP Control Plane and FRR
stay) and all namespaces; with tag21-r6-infra, Gateway,
Issuer and certificates disappear, the GatewayClass is cluster-wide:
k delete ciliumbgpadvertisement tag21-lb
k delete ciliumbgppeerconfig tag21-peer
k delete ciliumbgpclusterconfig tag21-bgp
k delete ciliumloadbalancerippool tag21-pool
k delete gatewayclass tag21-nodeport
k --context west delete ns tag21-r8
k config set-context --current --namespace=default
for n in r1 r2 r3 r4 r5 r6 r6-infra r7 r8 ch; do
k delete ns tag21-$n --wait=false
doneCheckpoint
k get ns | grep -c tag21; k --context west get ns | grep -c tag21Expected (after about a minute): 0 and
0.
k get ciliumnodeconfigs -A
k get ccnp,ippools,ciliumbgpclusterconfigs,ciliumbgppeerconfigs,ciliumbgpadvertisementsExpected: No resources found twice.
ssh [email protected] 'ip -d link show cilium_vxlan 2>/dev/null' \
| grep -o 'dstport [0-9]*'Expected: dstport 8472
diff <(k -n kube-system get cm coredns -o jsonpath='{.data}') \
<(k create -f ~/ckne/tag21/r3-coredns-backup.yaml $do -o jsonpath='{.data}') \
&& echo IDENTISCHExpected: IDENTISCH
cilium bgp peers
ssh [email protected] "ip route | grep -c proto.bgp"Expected: only the header line from
cilium bgp peers, 0 routes on
router.
cilium status | grep -E 'Cilium:|Operator:|Hubble Relay:|ClusterMesh:'
cilium clustermesh status | grep 'nodes are connected'Expected: OK four times,
All 3 nodes are connected to all clusters.
k get pods -A | grep -vE 'Running|Completed'Expected: only the header line.
What stays for Appendix B: Gateway API CRDs and
gatewayAPI.enabled, cert-manager, kube-prometheus-stack in
tag19-mon, bgpControlPlane.enabled together
with FRR on router, ClusterMesh with west.
Exam Tip
- Narrow it down first, then repair: name versus IP (DNS?), one node versus all (data path?), status conditions versus an actual test (Gateway green, curl red). Every half that works rules out half of the causes.
- Hubble names the reason and destination of every drop –
--verdict DROPPEDplus--drop-reason-descor--protocol httpinstead of guessing from logs. No flow (exit 7) means the Service has no backend. - Selectors are the most common source of errors: namespace labels
(
kubernetes.io/metadata.name), port names, BGP Advertisement labels, the annotation on both sides of a global Service.
Enterprise Note
- Per-node overrides (
CiliumNodeConfig) belong in Git and in review; a drift check compares thecilium-dbgvalues of all agents, not just the ConfigMap. - CoreDNS changes via GitOps with a syntax check
(
coredns -conf) in the pipeline; alerts oncoredns_dns_responses_totalby rcode and on reload errors. - Drop alerts from Hubble metrics (Day 19), separated by reason:
POLICY_DENIEDafter a deployment is a rollout problem, not an attack. - BGP configuration with tests in staging; the upstream router reports “routes from AS 65010 = 0” as an alert.
- Practice game days like this regularly with the on-call team – runbooks grow out of the hints of this day.