Skip to content
Hands-on IT IT Drills

Day 21: Competition – Repairing the Broken Network

Goal: You find and fix eight planted network faults across the CKNE curriculum – against the clock, with proof before and after. · Duration: approx. 90 min · Prerequisite: Days 0–20

Briefing

  • All five curriculum domains: Core Infrastructure and CNI, Service Networking and DNS, Advanced Traffic Management, Network Security and Policy, Observability.
  • Eight rounds, each in its own namespace (tag21-r1 … tag21-r8). Round 1 (agent), round 3 (CoreDNS) and round 7 (BGP) plant the fault in the cluster itself – you roll it back.
  • Tools as in the exam: k get/describe, status conditions, cilium, cilium-dbg, hubble observe, curl, dig, ssh to nodes and router.
  • Every round ends with a proof command: broken before, fixed after.
  • New in this Competition: CiliumNodeConfig (Cilium values per node, they override the ConfigMap; k get ciliumnodeconfigs -A) and hubble observe --drop-reason-desc (filter drops by reason). Both are also in the cheat sheet.
Round  Fault                                    Tool
1      Pods on worker2 isolated                 cilium-dbg status, CiliumNodeConfig
2      Service without backends                 EndpointSlice, port names
3      wrong answer for a Service name          dig, Corefile
4      Default Deny eats DNS                    hubble observe, NetworkPolicy
5      app reports 403                          hubble --protocol http, CNP L7
6      HTTPS at the Gateway broken              curl --cacert, Route status
7      LoadBalancer IP dead from outside        cilium bgp, FRR
8      failover to west does not kick in        clustermesh, global annotation
mkdir -p ~/ckne/tag21 && cd ~/ckne/tag21

Rules of Play

  • Per round: run the setup commands of the game master, then start the clock. Do not study the manifest – in the exam, too, you only see it once you go looking.
  • Find the cause, fix it, prove it with one command. Only then stop the clock.
  • Fix the cause, not the symptom: whoever deletes a policy instead of correcting it, or moves Pods to a different node, gets zero points.
  • Only read the Hint once the target time has run out.

Every round except round 8 starts the same way (round 8 creates the namespaces in both clusters itself), here for round 1:

k create ns tag21-r1
k config set-context --current --namespace=tag21-r1

Round 1: Island worker2

Situation: After a maintenance window, teams report: Pods on worker2 no longer resolve names and cannot reach Pods on other nodes. All Cilium Pods are Running. Find the cause and restore the normal state.

Game master:

k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
  --replicas=2 --port=8080 -- /agnhost netexec --http-port=8080
k expose deployment echo --port=80 --target-port=8080
k run c1 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k run c2 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k wait --for=condition=Ready pod --all
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNodeConfig
metadata:
  name: worker2-tuning
  namespace: kube-system
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: worker2
  defaults:
    tunnel-port: "8473"
EOF
k -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=worker2
k -n kube-system rollout status ds/cilium

Target time: 6 min

Hint: Configuration does not only come from the ConfigMap cilium-config. The agent’s init container log reveals which sources it has read.

Solution

Symptoms and a first look at the agents:

k exec c2 -- curl -sS -m3 echo/hostname; echo " rc=$?"
cilium status | grep -E 'Cilium:|Hubble Relay:'
for n in worker1 worker2; do
  CP=$(k -n kube-system get pod -l k8s-app=cilium \
    --field-selector spec.nodeName=$n -o name)
  k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status \
    | grep 'Cluster health'
done

Expected (abridged):

curl: (28) Resolving timed out after 3002 milliseconds
command terminated with exit code 28
 rc=28
 /¯¯\__/¯¯\    Cilium:             1 errors
 \__/¯¯\__/    Hubble Relay:       2 errors
Cluster health:          3/4 reachable   (2026-11-19T02:29:20Z)   ...
Cluster health:          1/3 reachable   (2026-11-19T02:27:57Z)   ...

cilium-health checks every node through the tunnel: worker2 only reaches itself, the others do not reach worker2 (the 4 includes west/argo). Where does the agent on worker2 get its configuration from ($CP points to it after the loop)?

k -n kube-system logs $CP -c config | grep 'Got configuration' | grep -o 'lenC.*'
k get ciliumnodeconfigs -A
k -n kube-system get ciliumnodeconfig worker2-tuning -o jsonpath='{.spec.defaults}'
echo
ssh [email protected] 'ip -d link show cilium_vxlan 2>/dev/null' \
  | grep -o 'dstport [0-9]*'
ssh [email protected] 'ip -d link show cilium_vxlan 2>/dev/null' \
  | grep -o 'dstport [0-9]*'

Expected (abridged):

lenConfigPairs=188 configSource="{Kind:config-map Namespace:kube-system Name:cil...
lenConfigPairs=1 configSource="{Kind:cilium-node-config Namespace:kube-system ...
NAMESPACE     NAME             AGE
kube-system   worker2-tuning   50m
{"tunnel-port":"8473"}
dstport 8472
dstport 8473

A CiliumNodeConfig overrides values of the ConfigMap for selected nodes. Here worker2 encapsulates VXLAN on UDP 8473, all others on 8472 – the packets arrive, but nobody unpacks them. cilium config view does not show this, it only reads the ConfigMap. Repair: delete the override, restart the agent:

k -n kube-system delete ciliumnodeconfig worker2-tuning
k -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=worker2
k -n kube-system rollout status ds/cilium
ssh [email protected] 'ip -d link show cilium_vxlan 2>/dev/null' \
  | grep -o 'dstport [0-9]*'

Expected:

ciliumnodeconfig.cilium.io "worker2-tuning" deleted from kube-system namespace
pod "cilium-nwzn2" deleted from kube-system namespace
daemon set "cilium" successfully rolled out
dstport 8472

After one to two minutes (until the other agents have registered the node again):

k exec c2 -- curl -s -m3 echo/hostname; echo
for i in $(seq 8); do k exec c1 -- curl -s -m2 echo/hostname; echo; done \
  | sort | uniq -c

Expected:

echo-cc8555f69-x8r52
      2 echo-cc8555f69-574ns
      6 echo-cc8555f69-x8r52

Both backends answer, across the node boundary too. Whoever only restarts the Pods instead changes nothing: the fault sits in the node’s data path.

Save your progress? Sign up for free

Round 2: The Service without a target

Situation: In tag21-r2, two healthy notes-api Pods are running, readiness green. The Service notes still refuses every connection. Repair the Service without touching the Deployment.

Game master (namespace and context as above, here tag21-r2):

k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
      - name: notes-api
        image: ghcr.io/thomaszachmann/notes-api:0.5.1
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
        ports:
        - name: api
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: api
---
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  selector:
    app: notes
  ports:
  - name: http
    port: 80
    targetPort: http
EOF
k run c --image=curlimages/curl:8.17.0 --command -- sleep infinity
k rollout status deploy notes

Target time: 3 min

Hint: The EndpointSlice has addresses. Look at the column next to them.

Solution

k exec c -- curl -sS -m3 notes/version; echo " rc=$?"
k get endpointslices
k describe svc notes | grep -E 'TargetPort|Endpoints'
k get pods -l app=notes -o jsonpath='{.items[0].spec.containers[0].ports}'; echo
CIP=$(k get svc notes -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A2 $CIP

Expected:

curl: (7) Failed to connect to notes port 80 after 3 ms: Could not connect to server
command terminated with exit code 7
 rc=7
NAME          ADDRESSTYPE   PORTS     ENDPOINTS            AGE
notes-9mbxg   IPv4          <unset>   10.0.1.2,10.0.2.71   36m
TargetPort:               http/TCP
Endpoints:                10.0.1.2,10.0.2.71
[{"containerPort":8080,"name":"api","protocol":"TCP"}]
75   10.96.118.49:80/TCP       ClusterIP

A named targetPort is resolved per Pod via the name of the container port. No Pod has a port http – the slice knows the addresses, but no port. Cilium therefore programs the Service without a backend and rejects every connection immediately (exit 7, no Hubble flow).

k patch svc notes -p '{"spec":{"ports":[{"port":80,"targetPort":"api"}]}}'
k get endpointslices
k exec c -- curl -s -m3 notes/version; echo
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A2 $CIP

Expected:

service/notes patched
NAME          ADDRESSTYPE   PORTS   ENDPOINTS            AGE
notes-9mbxg   IPv4          8080    10.0.2.71,10.0.1.2   36m
0.5.1
75   10.96.118.49:80/TCP       ClusterIP      1 => 10.0.1.2:8080/TCP (active)
                                              2 => 10.0.2.71:8080/TCP (active)

The patch hits the port entry via its key port: 80; the name http is kept.

Save your progress? Sign up for free

Round 3: The wrong twin

Situation: In tag21-r3 there are api (new) and api-v1 (old). Clients that call api end up at the old service. Deployments and Services are correct.

Game master (backup first – it serves as proof at the end):

for d in api api-v1; do
  k create deployment $d --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
    --port=8080 -- /agnhost netexec --http-port=8080
  k expose deployment $d --port=80 --target-port=8080
done
k run c --image=nicolaka/netshoot:v0.16 -- sleep infinity
k -n kube-system get cm coredns -o yaml > r3-coredns-backup.yaml
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > r3-Corefile
R='rewrite name exact api.tag21-r3.svc.cluster.local'
sed -i "s/^    ready$/    ready\n    $R api-v1.tag21-r3.svc.cluster.local/" r3-Corefile
k -n kube-system create configmap coredns --from-file=Corefile=r3-Corefile $do \
  | k replace -f -

Wait about a minute until both CoreDNS Pods have reloaded.

Target time: 5 min

Hint: Compare the IP from DNS with k get svc. Whoever answers names in the cluster reads its rules from a ConfigMap.

Solution

k exec c -- curl -s -m3 api/hostname; echo
k get svc
k exec c -- dig api.tag21-r3.svc.cluster.local | grep -A1 'ANSWER SECTION'
k exec c -- curl -s -m3 $(k get svc api -o jsonpath='{.spec.clusterIP}')/hostname
echo

Expected:

api-v1-9cc68fb5b-8n6xb
NAME     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
api      ClusterIP   10.97.130.83   <none>        80/TCP    89s
api-v1   ClusterIP   10.99.19.37    <none>        80/TCP    88s
;; ANSWER SECTION:
api.tag21-r3.svc.cluster.local. 30 IN   A   10.99.19.37
api-98c78c9bb-nt8wx

By IP the right Pod answers, by name the wrong one: the fault lies in DNS, not in the Service. The answer carries the requested name, but the IP of api-v1 – typical of rewrite, which translates the name in the answer back. Check and correct the Corefile:

k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -n rewrite
k -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > Corefile
sed -i '/rewrite name exact api.tag21-r3/d' Corefile
k -n kube-system create configmap coredns --from-file=Corefile $do | k replace -f -

Expected:

7:    rewrite name exact api.tag21-r3.svc.cluster.local api-v1.tag21-r3.svc.cluster...
configmap/coredns replaced

After about a minute (each CoreDNS Pod reloads on its own, Day 8):

k -n kube-system logs -l k8s-app=kube-dns --since=2m --prefix | grep 'Reloading c'
k exec c -- dig +short api.tag21-r3.svc.cluster.local
k exec c -- curl -s -m3 api/hostname; echo
diff <(k -n kube-system get cm coredns -o jsonpath='{.data}') \
  <(k create -f r3-coredns-backup.yaml $do -o jsonpath='{.data}') && echo IDENTISCH

Expected (abridged):

[pod/coredns-7cd4959b5-87shd/coredns] [INFO] Reloading complete
[pod/coredns-7cd4959b5-nkb2d/coredns] [INFO] Reloading complete
10.97.130.83
api-98c78c9bb-nt8wx
IDENTISCH

Only when both Pods report “Reloading complete” is the repair done – until then, every second lookup still answers wrongly.

Save your progress? Sign up for free

Round 4: Silence after Default Deny

Situation: In tag21-r4, Default Deny applies in both directions. web is supposed to reach notes via the Service name, no other Pod. The policies are “done”, yet web only gets timeouts.

Game master:

k create deployment notes --image=ghcr.io/thomaszachmann/notes-api:0.5.1 \
  --port=8080 $do > r4-notes.yaml
k set env --local -f r4-notes.yaml NOTES_API_KEY=lab-key-123 -o yaml | k apply -f -
k expose deployment notes --port=80 --target-port=8080
k run web --image=curlimages/curl:8.17.0 -l app=web --command -- sleep infinity
k apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-to-notes
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: notes
    ports:
    - port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: notes-from-web
spec:
  podSelector:
    matchLabels:
      app: notes
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web
    ports:
    - port: 8080
EOF
k rollout status deploy notes

Target time: 4 min

Hint: Test once with the name, once with the ClusterIP. Hubble tells you the destination of the dropped packet.

Solution

k exec web -- curl -sS -m5 notes/version; echo " rc=$?"
k exec web -- curl -sS -m5 $(k get svc notes -o jsonpath='{.spec.clusterIP}')/version
echo
hubble observe -P --port-forward-port 0 --from-pod tag21-r4/web --verdict DROPPED \
  --last 2

Expected (abridged):

curl: (28) Resolving timed out after 5000 milliseconds
command terminated with exit code 28
 rc=28
0.5.1
tag21-r4/web:32987 (ID:95867) <> kube-system/coredns-7cd4959b5-nkb2d:53 (ID:70078)
  policy-verdict:none EGRESS DENIED (UDP)
tag21-r4/web:32987 (ID:95867) <> kube-system/coredns-7cd4959b5-nkb2d:53 (ID:70078)
  Policy denied DROPPED (UDP)

By IP it works – the rules for 8080 are right. What gets dropped is DNS. Which label does allow-dns expect, which one does the namespace have?

k describe netpol allow-dns | grep -A2 'To:'
k get ns kube-system --show-labels

Expected:

    To:
      NamespaceSelector: name=kube-system
      PodSelector: k8s-app=kube-dns
NAME          STATUS   AGE   LABELS
kube-system   Active   4h    kubernetes.io/metadata.name=kube-system

Nobody sets the label name; the only reliable one is kubernetes.io/metadata.name, which the API server gives every namespace:

k patch netpol allow-dns --type=json -p '[{"op":"replace",
 "path":"/spec/egress/0/to/0/namespaceSelector/matchLabels",
 "value":{"kubernetes.io/metadata.name":"kube-system"}}]'
k exec web -- curl -sS -m5 notes/version; echo
k run fremd --image=curlimages/curl:8.17.0 --rm -i --restart=Never --command -- \
  curl -sS -m3 notes/version; echo " rc=$?"

Expected (abridged):

networkpolicy.networking.k8s.io/allow-dns patched
0.5.1
curl: (28) Connection timed out after 3002 milliseconds
pod "fremd" deleted from tag21-r4 namespace
pod tag21-r4/fremd terminated (Error)
 rc=28

fremd now resolves the name, but does not get past notes – Default Deny still applies.

Save your progress? Sign up for free

Round 5: The order with a 403

Situation: The Pod shop in tag21-r5 calls four notes-api endpoints in a loop and reports errors in its log without saying which call fails. Allow exactly the missing call – nothing more.

Game master:

k create deployment notes --image=ghcr.io/thomaszachmann/notes-api:0.5.1 \
  --port=8080 $do > r5-notes.yaml
k set env --local -f r5-notes.yaml NOTES_API_KEY=lab-key-123 -o yaml | k apply -f -
k expose deployment notes --port=80 --target-port=8080
k run shop --image=curlimages/curl:8.17.0 -l app=shop --command -- sh -c '
while true; do
  for r in "GET /version" "GET /notes" "POST /notes" "GET /healthz"; do
    set -- $r
    c=$(curl -s -o /dev/null -w "%{http_code}" -m3 -X $1 \
      -H "X-API-Key: lab-key-123" -d "{\"text\":\"bestellung\"}" notes$2)
    [ "$c" -ge 400 ] && echo "$(date +%T) FEHLER $c"
  done
  sleep 5
done'
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: notes-l7
spec:
  endpointSelector:
    matchLabels:
      app: notes
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: shop
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/(version|healthz|notes)"
EOF
k rollout status deploy notes

Target time: 4 min

Hint: A 403 without a timeout comes from Envoy. Hubble shows dropped requests with method and URL.

Solution

k logs shop --tail=2
hubble observe -P --port-forward-port 0 -n tag21-r5 --verdict DROPPED \
  --protocol http --last 2

Expected (abridged):

02:38:08 FEHLER 403
02:38:13 FEHLER 403
tag21-r5/shop:37468 (ID:78769) -> tag21-r5/notes-5f985975d8-p4k6r:8080 (ID:75232)
  http-request DROPPED (HTTP/1.1 POST http://notes/notes)
tag21-r5/shop:37498 (ID:78769) -> tag21-r5/notes-5f985975d8-p4k6r:8080 (ID:75232)
  http-request DROPPED (HTTP/1.1 POST http://notes/notes)

Only POST /notes is missing. Append a rule instead of loosening the existing one (method: ".*" would also open DELETE):

k patch cnp notes-l7 --type=json -p '[{"op":"add",
 "path":"/spec/ingress/0/toPorts/0/rules/http/-",
 "value":{"method":"POST","path":"/notes"}}]'
k get cnp notes-l7 -o jsonpath='{.spec.ingress[0].toPorts[0].rules.http}'; echo

Expected:

ciliumnetworkpolicy.cilium.io/notes-l7 patched
[{"method":"GET","path":"/(version|healthz|notes)"},{"method":"POST","path":"/notes"}]

After 15 seconds:

k logs shop --since=12s | wc -l
k exec shop -- curl -s -m3 -X DELETE notes/notes; echo
hubble observe -P --port-forward-port 0 --to-pod tag21-r5/notes --protocol http \
  --http-method POST --last 1

Expected (abridged):

0
Access denied
tag21-r5/shop:46614 (ID:78769) -> tag21-r5/notes-5f985975d8-p4k6r:8080 (ID:75232)
  http-request FORWARDED (HTTP/1.1 POST http://notes/notes)

No more errors in the log, POST works, DELETE stays forbidden.

Save your progress? Sign up for free

Round 6: The payment Gateway

Situation: The platform team runs the Gateway edge in tag21-r6-infra with a certificate from its own CA. The app team has created the route pay in tag21-r6. https://pay.example.com:<NodePort>/ is supposed to answer cleanly with --cacert. The route stays in the app namespace.

Game master:

k create ns tag21-r6-infra
k apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: tag21-r6-infra
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lab-ca
  namespace: tag21-r6-infra
spec:
  isCA: true
  commonName: Tag21 CA
  secretName: lab-ca
  privateKey:
    algorithm: ECDSA
  issuerRef:
    name: selfsigned
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: lab-ca
  namespace: tag21-r6-infra
spec:
  ca:
    secretName: lab-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: pay
  namespace: tag21-r6-infra
spec:
  secretName: pay-tls
  dnsNames:
  - shop.example.com
  issuerRef:
    name: lab-ca
---
apiVersion: cilium.io/v2alpha1
kind: CiliumGatewayClassConfig
metadata:
  name: nodeport
  namespace: tag21-r6-infra
spec:
  service:
    type: NodePort
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: tag21-nodeport
spec:
  controllerName: io.cilium/gateway-controller
  parametersRef:
    group: cilium.io
    kind: CiliumGatewayClassConfig
    name: nodeport
    namespace: tag21-r6-infra
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: edge
  namespace: tag21-r6-infra
spec:
  gatewayClassName: tag21-nodeport
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    hostname: pay.example.com
    tls:
      mode: Terminate
      certificateRefs:
      - name: pay-tls
    allowedRoutes:
      namespaces:
        from: Same
EOF
k create deployment pay --image=registry.k8s.io/gateway-api/echo-basic:v1.5.1
k expose deployment pay --port=8080 --target-port=3000
k apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: pay
  namespace: tag21-r6
spec:
  parentRefs:
  - name: edge
    namespace: tag21-r6-infra
    sectionName: https
  hostnames:
  - pay.example.com
  rules:
  - backendRefs:
    - name: pay
      port: 8080
EOF
k -n tag21-r6-infra wait --for=condition=Programmed gateway edge

For testing (part of the task, not of the fault):

NPS=$(k -n tag21-r6-infra get svc cilium-gateway-edge \
  -o jsonpath='{.spec.ports[0].nodePort}')
k -n tag21-r6-infra get secret lab-ca -o jsonpath='{.data.ca\.crt}' | base64 -d \
  > r6-ca.crt
R="--resolve pay.example.com:$NPS:172.16.0.228"

Target time: 6 min

Hint: Two faults in a row. The first is in the certificate itself, the second in the status of the route.

Solution

k -n tag21-r6-infra get gateway edge
curl -sS $R --cacert r6-ca.crt https://pay.example.com:$NPS/; echo "exit=$?"
k -n tag21-r6-infra get secret pay-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -ext subjectAltName

Expected (abridged):

NAME   CLASS            ADDRESS       PROGRAMMED   AGE
edge   tag21-nodeport   172.16.0.83   True         3s
curl: (60) SSL: no alternative certificate subject name matches target host name
'pay.example.com'
exit=60
X509v3 Subject Alternative Name: critical
    DNS:shop.example.com

Gateway Programmed, listener ResolvedRefs – as far as Kubernetes is concerned, everything is valid. Only the name in the certificate does not match the listener. The certificate belongs to cert-manager, so correct it there, not in the Secret:

k -n tag21-r6-infra patch certificate pay --type=merge \
  -p '{"spec":{"dnsNames":["pay.example.com"]}}'

Wait a few seconds until cert-manager has reissued it, then check:

k -n tag21-r6-infra get secret pay-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -ext subjectAltName
curl -s $R --cacert r6-ca.crt https://pay.example.com:$NPS/ -w '%{http_code}\n'
RC='{range .status.parents[*].conditions[*]}{.type}={.status} '\
'{.reason}: {.message}{"\n"}{end}'
k get httproute pay -o jsonpath="$RC"

Expected:

certificate.cert-manager.io/pay patched
X509v3 Subject Alternative Name: critical
    DNS:pay.example.com
404
Accepted=False NotAllowedByListeners: HTTPRoute is not allowed to attach to this
Gateway due to namespace restrictions
ResolvedRefs=True ResolvedRefs: Service reference is valid

TLS works, but Envoy has no route for the host (404). The listener only accepts routes from its own namespace. Allow exactly the app namespace, not All:

k -n tag21-r6-infra patch gateway edge --type=json -p '[{"op":"replace",
 "path":"/spec/listeners/0/allowedRoutes/namespaces",
 "value":{"from":"Selector","selector":{"matchLabels":
 {"kubernetes.io/metadata.name":"tag21-r6"}}}}]'
k get httproute pay -o jsonpath="$RC" | cut -d: -f1
k -n tag21-r6-infra get gateway edge -o jsonpath='{.status.listeners[0].attachedRoutes}'
echo
curl -s $R --cacert r6-ca.crt https://pay.example.com:$NPS/ | grep '"host"'

Expected:

gateway.gateway.networking.k8s.io/edge patched
Accepted=True Accepted
ResolvedRefs=True ResolvedRefs
1
 "host": "pay.example.com:32668",

A ReferenceGrant would not have helped here: it governs references from the route to foreign objects; attaching to a foreign Gateway is governed by allowedRoutes.

Save your progress? Sign up for free

Round 7: The invisible Service IP

Situation: The LoadBalancer Service echo in tag21-r7 has an IP from 10.50.0.0/24. It is supposed to be reachable from router via BGP, and according to the network team the sessions are up. Still: timeout.

Game master:

k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
  --port=8080 -- /agnhost netexec --http-port=8080
k expose deployment echo --type=LoadBalancer --port=80 --target-port=8080 \
  --labels=bgp=on
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: tag21-pool
spec:
  allowFirstLastIPs: "No"
  blocks:
  - cidr: 10.50.0.0/24
  serviceSelector:
    matchLabels:
      io.kubernetes.service.namespace: tag21-r7
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: tag21-bgp
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
  bgpInstances:
  - name: ost
    localASN: 65010
    peers:
    - name: router
      peerASN: 65000
      peerAddress: 172.16.0.123
      peerConfigRef:
        name: tag21-peer
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: tag21-peer
spec:
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: tag21
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: tag21-lb
  labels:
    advertise: bgp
spec:
  advertisements:
  - advertisementType: Service
    service:
      addresses:
      - LoadBalancerIP
    selector:
      matchLabels:
        bgp: "on"
EOF
k rollout status deploy echo

Target time: 5 min

Hint: A session being “established” does not mean “announces something”. Follow the chain ClusterConfig → PeerConfig → Advertisement via labels.

Solution

k get svc echo
ssh [email protected] curl -sS -m3 http://10.50.0.1/hostname; echo " rc=$?"
ssh [email protected] ip route get 10.50.0.1
cilium bgp peers
cilium bgp routes advertised ipv4 unicast

Expected (abridged):

NAME   TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
echo   LoadBalancer   10.99.60.2   10.50.0.1     80:30378/TCP   21s
curl: (28) Connection timed out after 3002 milliseconds
 rc=28
10.50.0.1 via 172.16.0.1 dev eth0 src 172.16.0.123 uid 1000
Node      Local AS   Peer AS   Peer Address   Session State   ...  Received   Advertised
cp1       65010      65000     172.16.0.123   established     ...  0          0
worker1   65010      65000     172.16.0.123   established     ...  0          0
worker2   65010      65000     172.16.0.123   established     ...  0          0
Node   VRouter   Peer   Prefix   NextHop   Age   Attrs

IP assigned, sessions up, but nothing announced – router sends 10.50.0.1 to its default gateway. What is the PeerConfig looking for, and what is the Advertisement labeled?

k get ciliumbgppeerconfig tag21-peer \
  -o jsonpath='{.spec.families[0].advertisements}'; echo
k get ciliumbgpadvertisements --show-labels

Expected:

{"matchLabels":{"advertise":"tag21"}}
NAME       AGE   LABELS
tag21-lb   29s   advertise=bgp

The PeerConfig’s selector finds no Advertisement. Set the label to match:

k label ciliumbgpadvertisement tag21-lb advertise=tag21 --overwrite
cilium bgp routes advertised ipv4 unicast
ssh [email protected] ip route show 10.50.0.1
ssh [email protected] curl -s -m3 http://10.50.0.1/hostname; echo

Expected (abridged):

ciliumbgpadvertisement.cilium.io/tag21-lb labeled
Node      VRouter   Peer           Prefix         NextHop        Age   Attrs
cp1       65010     172.16.0.123   10.50.0.1/32   172.16.0.126   9s    [{Origin: i} ...
worker1   65010     172.16.0.123   10.50.0.1/32   172.16.0.228   9s    [{Origin: i} ...
worker2   65010     172.16.0.123   10.50.0.1/32   172.16.0.83    9s    [{Origin: i} ...
10.50.0.1 nhid 85 proto bgp metric 20
    nexthop via 172.16.0.83 dev eth0 weight 1
    nexthop via 172.16.0.126 dev eth0 weight 1
    nexthop via 172.16.0.228 dev eth0 weight 1
echo-cc8555f69-4p8ld

Three equal-cost paths via ECMP, router reaches the Service.

Save your progress? Sign up for free

Round 8: Failover into the void

Situation: inventory runs in ost and west, namespace tag21-r8. According to the operations manual, west takes over when ost has no backends – which is why ost is scaled to 0 for maintenance. The client in ost still gets errors. Without scaling ost back up: restore the failover.

Game master (namespace in both clusters):

for c in ost west; do
  k --context $c create ns tag21-r8
  k --context $c -n tag21-r8 create deployment inventory-$c \
    --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --port=8080 \
    -- /agnhost netexec --http-port=8080
  k --context $c -n tag21-r8 expose deployment inventory-$c --name=inventory \
    --port=80 --target-port=8080
done
k config set-context --current --namespace=tag21-r8
k run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k annotate svc inventory service.cilium.io/global=true \
  service.cilium.io/affinity=local
k --context west -n tag21-r8 rollout status deploy inventory-west
k wait --for=condition=Ready pod/client
k scale deployment inventory-ost --replicas=0

Target time: 4 min

Hint: A global Service has two halves. Check both clusters.

Solution

k exec client -- curl -sS -m3 inventory/hostname; echo " rc=$?"
cilium clustermesh status | grep -E 'west:|nodes are connected'
k get svc inventory -o jsonpath='{.metadata.annotations}'; echo
k --context west -n tag21-r8 get svc inventory -o jsonpath='{.metadata.annotations}'
echo
CIP=$(k get svc inventory -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  --clustermesh-affinity | grep -A1 $CIP

Expected (abridged, without the CLI’s symbols):

curl: (7) Failed to connect to inventory:80 after 29 ms: Could not connect to server
command terminated with exit code 7
 rc=7
All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
  - west: 3/3 configured, 3/3 connected - KVStoreMesh: 1/1 configured, 1/1 connected
{"service.cilium.io/affinity":"local","service.cilium.io/global":"true"}

88   10.103.255.118:80/TCP     ClusterIP

The mesh is up, ost is global – but the Service in west has no annotation and does not share its backends. So there is no backend at all, exit 7.

k --context west -n tag21-r8 annotate svc inventory service.cilium.io/global=true
k exec client -- curl -s -m3 inventory/hostname; echo
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  --clustermesh-affinity | grep -A1 $CIP

Expected:

service/inventory annotated
inventory-west-66db64f486-dr5ss
88   10.103.255.118:80/TCP     ClusterIP      1 => 10.1.0.201:8080/TCP (active) ...

The backend from 10.1.0.0/16 (west) is there. End of maintenance – with affinity=local, traffic returns on its own:

k scale deployment inventory-ost --replicas=1
k rollout status deployment inventory-ost
for i in 1 2 3 4; do k exec client -- curl -s -m3 inventory/hostname; echo; done \
  | sort | uniq -c

Expected:

deployment.apps/inventory-ost scaled
deployment "inventory-ost" successfully rolled out
      4 inventory-ost-976947bf5-hn8hh

Stop the clock.

Save your progress? Sign up for free

Scoring

Round Fault Target time Points
1 CiliumNodeConfig with wrong tunnel port 6 min 3
2 named targetPort without a match 3 min 2
3 CoreDNS rewrite to old Service 5 min 2
4 DNS exception with wrong namespace label 4 min 2
5 L7 rule without POST /notes 4 min 2
6 wrong SAN + allowedRoutes Same 6 min 3
7 Advertisement label does not match the PeerConfig 5 min 2
8 west Service not global 4 min 2

You only get full points within the target time, without a hint and with a proof command; otherwise half (round down). Whoever removes the symptom instead of the cause (deleting a policy, moving Pods, allowedRoutes: All, method: ".*") gets 0 for the round. Overall target time: 37 min, maximum: 18 points.

Points Assessment
15–18 exam-ready – move on to the Practice Exam (Appendix B)
10–14 good; repeat the days of the rounds you missed
0–9 work through Days 4, 8, 10 and 12 once more

Challenge

Somewhere in the cluster, a policy drops traffic between two Pods. Find source, destination and policy using Hubble only, without knowing the namespace, and remove the leftover. Game master:

k create ns tag21-ch
k -n tag21-ch create deployment api \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --port=8080 \
  -- /agnhost netexec --http-port=8080
k -n tag21-ch expose deployment api --port=80 --target-port=8080
k -n tag21-ch run batch --image=curlimages/curl:8.17.0 -l app=batch --command -- \
  sh -c 'while true; do curl -s -m2 -o /dev/null api/hostname; sleep 3; done'
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: tag21-ch-altlast
spec:
  endpointSelector:
    matchLabels:
      app: api
  enableDefaultDeny:
    ingress: false
  ingressDeny:
  - fromEndpoints:
    - matchLabels:
        app: batch
EOF

Hint: hubble observe --verdict DROPPED shows all drops – in the author’s lab mostly VLAN broadcasts from the home network (Day 0). --drop-reason-desc filters by reason; the policy reasons are called POLICY_DENIED and POLICY_DENY.

Solution

hubble observe -P --port-forward-port 0 --verdict DROPPED --last 2
hubble observe -P --port-forward-port 0 --verdict DROPPED \
  --drop-reason-desc POLICY_DENY --drop-reason-desc POLICY_DENIED --since 5m \
  | tail -2

Expected (abridged, depends on the network – without VLAN broadcasts, the two VLAN_FILTERED lines are missing for you, and the first command already shows the policy drops):

10.10.10.0:36526 (world) <> 10.10.11.255:10001 (world) VLAN traffic disallowed by
  VLAN filter DROPPED (UDP)
192.168.3.1:36526 (world) <> 192.168.3.255:10001 (world) VLAN traffic disallowed by
  VLAN filter DROPPED (UDP)
tag21-ch/batch:60336 (ID:71887) <> tag21-ch/api-98c78c9bb-vgt7r:8080 (ID:94484)
  policy-verdict:L3-Only INGRESS DENIED (TCP Flags: SYN)
tag21-ch/batch:60336 (ID:71887) <> tag21-ch/api-98c78c9bb-vgt7r:8080 (ID:94484)
  Policy denied by denylist DROPPED (TCP Flags: SYN)

“denied by denylist” means: an explicit deny rule, not Default Deny. There is none in the namespace – so search cluster-wide:

k -n tag21-ch get cnp,netpol
k get ccnp -o json | jq -r '.items[]
  | select(.spec.ingressDeny or .spec.egressDeny) | .metadata.name'
k delete ccnp tag21-ch-altlast
hubble observe -P --port-forward-port 0 --from-pod tag21-ch/batch --to-port 8080 \
  --last 1

Expected (abridged):

No resources found in tag21-ch namespace.
tag21-ch-altlast
ciliumclusterwidenetworkpolicy.cilium.io "tag21-ch-altlast" deleted
tag21-ch/batch:56220 (ID:71887) -> tag21-ch/api-98c78c9bb-vgt7r:8080 (ID:94484)
  to-endpoint FORWARDED (TCP Flags: ACK)

Save your progress? Sign up for free

Cleanup

The cluster changes from rounds 1 and 3 are already rolled back. What goes are the BGP objects from round 7 (the BGP Control Plane and FRR stay) and all namespaces; with tag21-r6-infra, Gateway, Issuer and certificates disappear, the GatewayClass is cluster-wide:

k delete ciliumbgpadvertisement tag21-lb
k delete ciliumbgppeerconfig tag21-peer
k delete ciliumbgpclusterconfig tag21-bgp
k delete ciliumloadbalancerippool tag21-pool
k delete gatewayclass tag21-nodeport
k --context west delete ns tag21-r8
k config set-context --current --namespace=default
for n in r1 r2 r3 r4 r5 r6 r6-infra r7 r8 ch; do
  k delete ns tag21-$n --wait=false
done

Checkpoint

k get ns | grep -c tag21; k --context west get ns | grep -c tag21

Expected (after about a minute): 0 and 0.

k get ciliumnodeconfigs -A
k get ccnp,ippools,ciliumbgpclusterconfigs,ciliumbgppeerconfigs,ciliumbgpadvertisements

Expected: No resources found twice.

ssh [email protected] 'ip -d link show cilium_vxlan 2>/dev/null' \
  | grep -o 'dstport [0-9]*'

Expected: dstport 8472

diff <(k -n kube-system get cm coredns -o jsonpath='{.data}') \
  <(k create -f ~/ckne/tag21/r3-coredns-backup.yaml $do -o jsonpath='{.data}') \
  && echo IDENTISCH

Expected: IDENTISCH

cilium bgp peers
ssh [email protected] "ip route | grep -c proto.bgp"

Expected: only the header line from cilium bgp peers, 0 routes on router.

cilium status | grep -E 'Cilium:|Operator:|Hubble Relay:|ClusterMesh:'
cilium clustermesh status | grep 'nodes are connected'

Expected: OK four times, All 3 nodes are connected to all clusters.

k get pods -A | grep -vE 'Running|Completed'

Expected: only the header line.

What stays for Appendix B: Gateway API CRDs and gatewayAPI.enabled, cert-manager, kube-prometheus-stack in tag19-mon, bgpControlPlane.enabled together with FRR on router, ClusterMesh with west.

Exam Tip

  • Narrow it down first, then repair: name versus IP (DNS?), one node versus all (data path?), status conditions versus an actual test (Gateway green, curl red). Every half that works rules out half of the causes.
  • Hubble names the reason and destination of every drop – --verdict DROPPED plus --drop-reason-desc or --protocol http instead of guessing from logs. No flow (exit 7) means the Service has no backend.
  • Selectors are the most common source of errors: namespace labels (kubernetes.io/metadata.name), port names, BGP Advertisement labels, the annotation on both sides of a global Service.

Enterprise Note

  • Per-node overrides (CiliumNodeConfig) belong in Git and in review; a drift check compares the cilium-dbg values of all agents, not just the ConfigMap.
  • CoreDNS changes via GitOps with a syntax check (coredns -conf) in the pipeline; alerts on coredns_dns_responses_total by rcode and on reload errors.
  • Drop alerts from Hubble metrics (Day 19), separated by reason: POLICY_DENIED after a deployment is a rollout problem, not an attack.
  • BGP configuration with tests in staging; the upstream router reports “routes from AS 65010 = 0” as an alert.
  • Practice game days like this regularly with the on-call team – runbooks grow out of the hints of this day.