Skip to content
Hands-on IT IT Drills

Day 14: Competition – Disaster Drills

Goal: Find and fix eight typical network and admission failures under time pressure without tearing open new holes. · Duration: approx. 180 min · Prerequisite: Day 13 (Drill 8 additionally Day 4)

Briefing

  • Every drill has a trigger. Fire it, then start the stopwatch and only stop it once the competition matrix is green again.
  • Measure first, then change: np-matrix.sh, conncheck.sh, events, reports, logs. Whoever guesses loses time.
  • Read only the scenario and symptoms. Uncover the diagnosis path and solution only when you are stuck. That costs half the points.
  • Drills 1–7 run on kyverno-lab, Drill 8 on cilium-lab.
  • None of the cluster-related outputs have been tested live. Checked live are the Kyverno CLI 1.19.1 (Drills 3, 7, challenge) and the syntax of all files.
 failure ──► symptom (matrix ✘, event, log) ──► diagnosis ──► smallest fix
                                                     └──► lasting lesson (Git, policy)

Drill 0: Lineup (untimed)

A defined starting state: registry and lab CA from Day 10, Kyverno HA from Day 12, policies from Days 8 to 13. Everything comes fresh from ~/guard-lab/code.

mkdir -p ~/guard-lab/tag14 && cd ~/guard-lab/tag14
kubectl config use-context kind-kyverno-lab
alias cc=~/guard-lab/code/demo/conncheck.sh
cp -R ~/guard-lab/code/kyverno-ops/tag13/{policies,tests} .
(cd ~/guard-lab/tag10 && ./setup-registry.sh)
ARCH=$(kubectl get node -o jsonpath='{.items[0].status.nodeInfo.architecture}')
crane copy --insecure --platform linux/$ARCH \
  nginxinc/nginx-unprivileged:1.31-alpine localhost:5001/shop/web:1.31 2>/dev/null
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.9.1 \
  -f ~/guard-lab/tag12/values-ha.yaml \
  --set-file global.caCertificates.data=$HOME/guard-lab/tag10/certs/registry.crt \
  --wait --timeout 5m
kubectl -n kyverno rollout restart deploy kyverno-admission-controller \
  kyverno-background-controller kyverno-reports-controller
kubectl -n kyverno rollout status deploy kyverno-admission-controller --timeout=180s

Expected (abridged, not tested live, may differ depending on version):

✔ kyverno-lab-control-plane vertraut kind-registry:5000
✔ Registry erreichbar: https://localhost:5001
Release "kyverno" has been upgraded. Happy Helming!
deployment "kyverno-admission-controller" successfully rolled out

Why the CA again? values-ha.yaml from Day 12 does not contain it. Every helm upgrade -f values-ha.yaml without --set-file silently throws it out, and Day 10 ends with x509: certificate signed by unknown authority.

Now the policies and the template for monitoring:

K=~/guard-lab/code/kyverno
kubectl apply -f $K/tag08/image-hygiene.yaml \
  -f ~/guard-lab/tag10/verify-shop-images.yaml \
  -f ~/guard-lab/tag11/policy-repo/policies/exception-governance.yaml \
  -f ~/guard-lab/tag12/no-privileged-vap.yaml \
  -f $K/tag09/tenant-netpol.yaml -f policies/
kubectl patch gpol tenant-netpol --type merge \
  -p '{"spec":{"evaluation":{"generateExisting":{"enabled":true}}}}'
kubectl create namespace netpol-templates
kubectl apply -f $K/tag09/tpl-allow-monitoring.yaml -f $K/tag09/clone-monitoring.yaml
sleep 20
kubectl get netpol -n tenant-a

Expected (abridged, not tested live, may differ depending on version):

NAME                   POD-SELECTOR   AGE
allow-dns              <none>         15s
allow-monitoring       tier=api       12s
allow-same-namespace   <none>         2d
default-deny           <none>         15s

The target matrix for the entire competition, checked with np-matrix.sh from Day 6:

cat > wettkampf.tsv <<'EOF'
# von                 nach                                 soll
default/client        http://frontend.shop                 allow
shop/frontend         http://api.shop                      allow
shop/api              tcp://db.shop:5432                   allow
tenant-a/api          http://api.tenant-a                  allow
tenant-a/api          http://api.tenant-b                  deny
tenant-b/api          http://api.tenant-a                  deny
default/client        http://api.tenant-a                  deny
monitoring/scraper    http://api.tenant-a:9113/metrics     allow
monitoring/scraper    http://api.tenant-b:9113/metrics     allow
monitoring/scraper    http://api.tenant-a                  deny
EOF
M=~/guard-lab/code/np-tests/np-matrix.sh
$M wettkampf.tsv | tail -1

Expected (abridged, not tested live, may differ depending on version):

10 Tests: 10 ✔, 0 ✘

Not green? Then the starting state is wrong. Fix it first, then start.

Warning: Work only in the lab. Several triggers are deliberately brutal (deleting webhooks, labels on platform namespaces). In production they belong in a game day protocol with approval.

Save your progress? Sign up for free

Drill 1: Egress hardening without DNS

Scenario: The shop team wants to “prevent data exfiltration” and only allows egress within the namespace. Two minutes later, support reports: orders are hanging.

Trigger:

cat > egress-lockdown.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-lockdown
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - podSelector: {}         # "only within our own namespace"
EOF
kubectl apply -f egress-lockdown.yaml; sleep 15

Time limit: 5 min

Symptoms:

$ kubectl exec client -- curl -s -o /dev/null -w '%{http_code}\n' http://frontend.shop/
200
$ kubectl exec client -- curl -s -o /dev/null -w '%{http_code}\n' \
    http://frontend.shop/api/orders
502

Diagnosis path:

$M wettkampf.tsv | grep '✘'
kubectl -n shop logs deploy/frontend --tail=3 | grep -o 'could not be resolved.*'
kubectl -n shop exec deploy/frontend -- nslookup api.shop.svc.cluster.local
kubectl -n shop get netpol
kubectl -n shop describe netpol egress-lockdown | sed -n '/Spec/,$p'

Expected (abridged, not tested live, may differ depending on version):

shop/frontend          http://api.shop                    allow  ✘ deny
could not be resolved (110: Operation timed out)
;; connection timed out; no servers could be reached
NAME              POD-SELECTOR   AGE
egress-lockdown   <none>         1m
  Allowing egress traffic:
    To Port: <any> (traffic allowed to all ports)
    To:
      PodSelector: <none>

The frontend is alive, only /api/ fails: nginx re-resolves api.shop every 10 s. CoreDNS lives in kube-system, and egress there is now closed (Day 2, Drill 1).

Solution:

kubectl apply -f ~/guard-lab/tag02/allow-dns.yaml
sleep 5
$M wettkampf.tsv | tail -1

Expected (abridged, not tested live):

networkpolicy.networking.k8s.io/allow-dns created
10 Tests: 10 ✔, 0 ✘

The hardening stays, DNS is added on top. Do not delete the policy; the team wanted it.

Lessons:

  • Egress default deny and allow-dns are a pair. Never roll out one without the other.
  • “Page loads, feature broken” is the typical picture: ingress ok, egress DNS gone.
  • Here conncheck.sh reports “Timeout”, not “DNS-Fehler” (DNS error): The query starves.

Save your progress? Sign up for free

Drill 2: Kyverno gone, failurePolicy Fail

Scenario: A colleague tests a Kyverno update by hand, “real quick”. Afterwards no deployment goes through anymore, and scaling back up does not help.

Trigger:

kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=0
kubectl -n kyverno set image deploy/kyverno-admission-controller \
  kyverno=reg.kyverno.io/kyverno/kyverno:v1.19.99

Time limit: 10 min

Symptoms:

$ kubectl -n shop rollout restart deploy frontend
error: failed to patch: Internal error occurred: failed calling webhook
"…validate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc
.kyverno.svc:443/…": no endpoints available for service "kyverno-svc"

Diagnosis path:

kubectl -n kyverno get deploy kyverno-admission-controller
kubectl -n kyverno get endpointslice -l kubernetes.io/service-name=kyverno-svc
kubectl get validatingwebhookconfigurations -l webhook.kyverno.io/managed-by=kyverno \
  -o custom-columns=NAME:.metadata.name,POLICY:.webhooks[*].failurePolicy
kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=3
sleep 20
kubectl -n kyverno get pods -l app.kubernetes.io/component=admission-controller

Expected (abridged, not tested live, may differ depending on version):

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
kyverno-admission-controller   0/0     0            0           3d
NAME                ADDRESSTYPE   PORTS     ENDPOINTS   AGE
kyverno-svc-xxxxx   IPv4          <unset>   <unset>     3d
NAME                                      POLICY
kyverno-resource-validating-webhook-cfg   Fail,Ignore,Fail
...
kyverno-admission-controller-7f9c…-4kq2p   0/1   ImagePullBackOff   0   20s

Webhooks registered, no endpoint, the new image does not exist. Scaling alone heals nothing.

Solution: First the emergency exit, then the cause, then the proof.

kubectl -n shop run priv --image=nginxinc/nginx-unprivileged:1.31-alpine \
  --privileged --dry-run=server -o name 2>&1 | tail -1
kubectl delete validatingwebhookconfiguration,mutatingwebhookconfiguration \
  -l webhook.kyverno.io/managed-by=kyverno
kubectl -n shop rollout restart deploy frontend
kubectl -n kyverno rollout undo deploy kyverno-admission-controller
kubectl -n kyverno rollout status deploy kyverno-admission-controller --timeout=180s
sleep 15
kubectl get validatingwebhookconfigurations -l webhook.kyverno.io/managed-by=kyverno \
  --no-headers | wc -l
kubectl -n shop run x -l app=x --image=nginxinc/nginx-unprivileged:latest \
  --dry-run=server -o name 2>&1 | sed 's/.*denied the request: //'

Expected (abridged, not tested live, may differ depending on version):

The pods "priv" is invalid: : ValidatingAdmissionPolicy 'no-privileged' with binding
'no-privileged-binding' denied request: Privilegierte Container sind verboten.
validatingwebhookconfiguration.admissionregistration.k8s.io "kyverno-…" deleted
...
deployment.apps/frontend restarted
deployment.apps/kyverno-admission-controller rolled back
deployment "kyverno-admission-controller" successfully rolled out
4
Policy image-hygiene failed: Fester Tag oder Digest nötig, nicht :latest:
nginxinc/nginx-unprivileged:latest

The first line is the good news: The native VAP from Day 12 works even without Kyverno. rollout undo brings back the old image; the replicas are already at 3. Kyverno recreates its webhooks itself on startup.

Lessons:

  • Deleting webhooks is the emergency exit, not a fix. Until Kyverno runs again, only what lives in the API server as VAP or PSA applies. Record the time window in the incident log.
  • Never change Kyverno via kubectl set image. Versions only via Helm and Git (Day 12).
  • kube-system and kyverno are excluded. Otherwise the cluster locks itself out on restart, and the emergency exit is the only way.

Save your progress? Sign up for free

Drill 3: A GeneratingPolicy spreads a mistake

Scenario: The platform “simplifies” tenant-onboarding: allow-same-namespace should only govern ingress. Thanks to synchronize, the rollout reaches every tenant within seconds.

Trigger:

sed 's/"policyTypes": dyn(\["Ingress", "Egress"\]),/"policyTypes": dyn(["Ingress"]),/' \
  policies/tenant-onboarding.yaml > tenant-onboarding-v2.yaml
kubectl apply -f tenant-onboarding-v2.yaml; sleep 20

Time limit: 10 min

Symptoms: Traffic within all tenants breaks down; between tenants everything stays sealed. A manual repair only lasts seconds.

tenant-a/api           http://api.tenant-a                allow  ✘ deny
10 Tests: 9 ✔, 1 ✘

Diagnosis path:

kubectl -n tenant-a describe netpol allow-same-namespace | grep -E 'egress|Types'
kubectl get netpol -A -l generate.kyverno.io/policy-name=tenant-onboarding \
  -o custom-columns=NS:.metadata.namespace,TYPES:.spec.policyTypes
diff policies/tenant-onboarding.yaml tenant-onboarding-v2.yaml
kyverno apply tenant-onboarding-v2.yaml \
  -r ~/guard-lab/code/kyverno-ops/tag13/tenant-d.yaml --remove-color \
  | sed -n '/allow-same-namespace/,/^---/p' | grep -A1 policyTypes

Expected (diff and kyverno apply tested live, rest abridged):

  Not affecting egress traffic
  Policy Types: Ingress
NS         TYPES
tenant-a   ["Ingress"]
tenant-b   ["Ingress"]
tenant-d   ["Ingress"]
36c36
<             "policyTypes": dyn(["Ingress", "Egress"]),
---
>             "policyTypes": dyn(["Ingress"]),
    policyTypes:
        - Ingress

The egress rule is still in the object, but without Egress in policyTypes it is ignored. default-deny thus also blocks traffic within the namespace.

Solution: Fix the source, not the copies.

kubectl -n tenant-a patch netpol allow-same-namespace --type merge \
  -p '{"spec":{"policyTypes":["Ingress","Egress"]}}'; sleep 10
kubectl -n tenant-a get netpol allow-same-namespace -o jsonpath='{.spec.policyTypes}'
echo
kubectl apply -f policies/tenant-onboarding.yaml; sleep 20
$M wettkampf.tsv | tail -1

Expected (abridged, not tested live, may differ depending on version):

networkpolicy.networking.k8s.io/allow-same-namespace patched
["Ingress"]
generatingpolicy.policies.kyverno.io/tenant-onboarding configured
10 Tests: 10 ✔, 0 ✘

The hotfix in the namespace is reverted (Day 9, Drill 4). Only the corrected policy spreads everywhere again.

Lessons:

  • synchronize spreads mistakes just as fast as corrections. Generated policies need the same tests as handwritten ones: kyverno apply against a sample namespace shows the result before the merge.
  • During an incident, always fix the policy in Git. Hand patches to generated objects are lost minutes.
  • A matrix case “allowed within the tenant” belongs in every target matrix, not just the denials.

Save your progress? Sign up for free

Drill 4: Unsigned hotfix image during an incident

Scenario: A production bug in the shop. The hotfix is built, but the CI signing step has failed. verify-shop-images (Day 10) does not let the image through. The incident manager wants to see the hotfix live in 10 minutes.

Trigger:

crane copy --insecure localhost:5001/shop/web:1.31 localhost:5001/shop/hotfix:1.31 \
  2>/dev/null
kubectl -n shop create deploy hotfix --image=kind-registry:5000/shop/hotfix:1.31 \
  --dry-run=client -o yaml \
  | kubectl set resources --local -f - --requests=cpu=10m,memory=16Mi \
    --limits=memory=64Mi -o yaml > hotfix.yaml
kubectl apply -f hotfix.yaml

Time limit: 10 min

Symptoms:

Error from server: error when creating "hotfix.yaml": admission webhook "…" denied the
request: Policy verify-shop-images failed: Image ist nicht mit dem Lab-Key signiert.

Diagnosis path:

kubectl -n kyverno logs deploy/kyverno-admission-controller --since=5m \
  | grep -oE 'no signatures found|x509: [a-z ]+' | sort | uniq -c
crane ls --insecure localhost:5001/shop/hotfix
kubectl get ivpol verify-shop-images -o jsonpath='{.spec.matchImageReferences}'
echo

Expected (abridged, not tested live, may differ depending on version):

      1 no signatures found
1.31
[{"glob":"kind-registry:5000/shop/*"}]

no signatures found means: The policy is correct, the image really is unsigned. If it says x509: certificate signed by unknown authority, Kyverno is missing the lab CA (Drill 0), and every image fails. That would be a different incident.

Solution: An exception, as narrow and as short as possible, with a ticket. Afterwards, sign the image retroactively and remove the exception immediately.

BIS=$(date -u -v+24H +%Y-%m-%dT%H:%M:%SZ)   # Linux: date -u -d '+24 hours' +%FT%TZ
cat > hotfix-exception.yaml <<EOF
# Day 14: emergency exception - unsigned hotfix image, 24 h, ticket mandatory
apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata:
  name: hotfix-web-unsigned
  namespace: policy-exceptions
spec:
  policyRefs:
  - name: verify-shop-images
    kind: ImageValidatingPolicy
  matchConditions:
  - name: nur-shop-hotfix
    expression: >-
      object.metadata.namespace == 'shop' &&
      object.metadata.?labels[?'app'].orValue('') == 'hotfix'
  expiresAt: "$BIS"
  properties:
    ticket: INC-0815
    owner: team-shop
    reason: Hotfix ohne Signatur, CI-Signierschritt ausgefallen
EOF
kyverno apply ~/guard-lab/tag11/policy-repo/policies/exception-governance.yaml \
  -r hotfix-exception.yaml --remove-color | tail -1
kubectl apply -f hotfix-exception.yaml
kubectl apply -f hotfix.yaml
kubectl -n shop rollout status deploy hotfix --timeout=90s

Expected (first line tested live, rest abridged, not tested live):

pass: 1, fail: 0, warn: 0, error: 0, skip: 0
policyexception.policies.kyverno.io/hotfix-web-unsigned created
deployment.apps/hotfix created
deployment "hotfix" successfully rolled out

After the incident, you add the signature retroactively (key and offline.json from Day 10) and prove that it works without the exception:

cd ~/guard-lab/tag10
D=$(crane digest --insecure localhost:5001/shop/hotfix:1.31)
cosign sign --registry-cacert certs/registry.crt --key cosign.key \
  --signing-config offline.json -y "localhost:5001/shop/hotfix@$D"
cosign attest --registry-cacert certs/registry.crt --key cosign.key \
  --signing-config offline.json -y --type spdxjson --predicate sbom.spdx.json \
  "localhost:5001/shop/hotfix@$D"
cd ~/guard-lab/tag14
kubectl -n policy-exceptions delete policyexceptions.policies.kyverno.io \
  hotfix-web-unsigned
kubectl -n shop rollout restart deploy hotfix
kubectl -n shop rollout status deploy hotfix --timeout=90s

Expected (abridged, not tested live, may differ depending on version):

policyexception.policies.kyverno.io "hotfix-web-unsigned" deleted
deployment.apps/hotfix restarted
deployment "hotfix" successfully rolled out

Why the SBOM too? verify-shop-images checks the signature and the SPDX attestation (Day 10). The SBOM from Day 10 fits, because the digest is the same.

Lessons:

  • During an incident, an exception is the right tool; switching off the policy never is. exception-governance (Day 11) enforces a ticket, an owner and an expiration.
  • Cut it narrow: one policy, one namespace, one label. expiresAt in hours, not in months. delete-expired-exceptions cleans up the rest (Day 12).
  • Check the Kyverno logs first: missing signature, wrong key, missing CA or registry gone are four different incidents with the same message to the client.

Save your progress? Sign up for free

Drill 5: Label drift opens the tenant boundary

Scenario: A team creates tenant-e and copies the manifest of the monitoring namespace for it. Nobody notices anything until a pentest fetches other tenants’ metrics from tenant-e.

Trigger:

cat > tenant-e.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-e
  labels: {guard-lab: demo, team: platform, purpose: monitoring}   # copied
EOF
kubectl apply -f tenant-e.yaml
sed 's/namespace: default/namespace: tenant-e/' ~/guard-lab/code/demo/40-client.yaml \
  | kubectl apply -f -
kubectl -n tenant-e wait --for=condition=Ready pod/client --timeout=60s

Time limit: 10 min

Symptoms:

$ cc tenant-e/client http://api.tenant-a:9113/metrics deny
✔ erlaubt   tenant-e/client -> http://api.tenant-a:9113/metrics (HTTP 200)
  <-- erwartet: deny

Diagnosis path:

kubectl get ns tenant-e --show-labels
kubectl get netpol -n tenant-e
kubectl -n tenant-a describe netpol allow-monitoring | sed -n '/Allowing/,/Not/p'
kubectl get ns -l purpose=monitoring

Expected (abridged, not tested live, may differ depending on version):

NAME       STATUS   AGE   LABELS
tenant-e   Active   2m    guard-lab=demo,kubernetes.io/metadata.name=tenant-e,…
No resources found in tenant-e namespace.
  Allowing ingress traffic:
    To Port: 9113/TCP
    From:
      NamespaceSelector: purpose=monitoring
NAME         STATUS   AGE
monitoring   Active   5d
tenant-e     Active   2m

Two mistakes interlock: Without the label tenant, tenant-e has no default deny (egress open). And with purpose=monitoring, it counts as monitoring in every tenant (Day 3, Drill 6: labels as a contract).

Solution: First get the labels right, then harden the template so that a wrong label alone is never enough again.

kubectl label ns tenant-e purpose- team=tenant-e tenant=e --overwrite
kubectl -n netpol-templates patch netpol allow-monitoring --type json -p '[{"op":
  "replace","path":"/spec/ingress/0/from","value":[{"namespaceSelector":{
  "matchLabels":{"kubernetes.io/metadata.name":"monitoring"}},
  "podSelector":{"matchLabels":{"app":"scraper"}}}]}]'
sleep 20
kubectl get netpol -n tenant-e
cc tenant-e/client http://api.tenant-a:9113/metrics deny
$M wettkampf.tsv | tail -1

Expected (abridged, not tested live, may differ depending on version):

namespace/tenant-e labeled
networkpolicy.networking.k8s.io/allow-monitoring patched
NAME                   POD-SELECTOR   AGE
allow-dns              <none>         15s
allow-monitoring       tier=api       15s
allow-same-namespace   <none>         15s
default-deny           <none>         15s
✘ blockiert tenant-e/client -> http://api.tenant-a:9113/metrics (Timeout)
10 Tests: 10 ✔, 0 ✘

The label tenant triggers generation; the changed template synchronizes into all tenants (Day 9, Drill 6). The new source is an AND of kubernetes.io/metadata.name and app=scraper, which no team can forge.

Lessons:

  • Copied manifests copy permissions along with them. Self-assigned labels like purpose are only as secure as the rule that protects them (challenge).
  • Always select foreign namespaces via kubernetes.io/metadata.name (Day 3).
  • A missing default deny is not an error that any policy reports. require-default-deny (Day 13) only checks namespaces with tenant. If the label is missing, the check is missing too.

Save your progress? Sign up for free

Drill 6: A rollout locks out monitoring

Scenario: The platform wants quota and Pod Security “for monitoring too” and attaches the tenant label to the namespace. Shortly afterwards, every alert fires TargetDown.

Trigger:

kubectl label ns monitoring tenant=platform
sleep 20

Time limit: 10 min

Symptoms:

$ kubectl -n monitoring logs deploy/scraper --tail=3
FAIL shop
FAIL tenant-a
FAIL tenant-b

Diagnosis path:

$M wettkampf.tsv | grep '✘'
kubectl get netpol,resourcequota,rolebinding -n monitoring \
  -L generate.kyverno.io/policy-name
kubectl get ns monitoring -o json | jq -c .metadata.labels

Expected (abridged, not tested live, may differ depending on version):

monitoring/scraper     http://api.tenant-a:9113/metrics   allow  ✘ deny
monitoring/scraper     http://api.tenant-b:9113/metrics   allow  ✘ deny
NAME                                           …   POLICY-NAME
networkpolicy…/allow-dns                       …   tenant-netpol
networkpolicy…/allow-monitoring                …   clone-monitoring
networkpolicy…/allow-same-namespace            …   tenant-onboarding
networkpolicy…/default-deny                    …   tenant-netpol
resourcequota/tenant-quota                     …   tenant-onboarding
rolebinding…/team-edit                         …   tenant-onboarding
{"guard-lab":"demo",…,"pod-security.kubernetes.io/enforce":"restricted",…,
"tenant":"platform"}

One label, six objects and three PSA labels. default-deny blocks the scraper’s egress, and team-edit casually gives the group platform edit in the namespace.

Solution: Removing the label is not enough. Generated objects and mutations stay in place (Day 9, Drill 7); you clean them up selectively:

kubectl label ns monitoring tenant- pod-security.kubernetes.io/enforce- \
  pod-security.kubernetes.io/enforce-version- pod-security.kubernetes.io/warn-
P=tenant-netpol,tenant-onboarding,clone-monitoring
kubectl -n monitoring delete netpol,resourcequota,rolebinding \
  -l "generate.kyverno.io/policy-name in ($P)"
sleep 15
kubectl -n monitoring logs deploy/scraper --tail=3
$M wettkampf.tsv | tail -1

Expected (abridged, not tested live, may differ depending on version):

namespace/monitoring unlabeled
networkpolicy.networking.k8s.io "allow-dns" deleted from monitoring namespace
...
rolebinding.rbac.authorization.k8s.io "team-edit" deleted from monitoring namespace
ok   shop
ok   tenant-a
ok   tenant-b
10 Tests: 10 ✔, 0 ✘

If the objects come back after deletion, the namespace still matches a GeneratingPolicy: check the labels again.

Lessons:

  • A trigger label is a rollout switch for everything that hangs off it: network, quota, RBAC, PSA. Before setting it, run kyverno apply against the namespace (Day 13, Drill 5) and read the list.
  • Platform namespaces are not tenants. If monitoring wants default deny, it gets its own policies with egress to all scrape targets.
  • The scraper log is the fastest sensor in the lab; in production, up == 0 per target.

Save your progress? Sign up for free

Drill 7: CI green, cluster red

Scenario: A pull request switches no-world-egress to the new label schema tenant-tier. kyverno test is green, review done, merge, rollout. A week later the audit finds 0.0.0.0/0 in three tenants.

Trigger:

ALT='- {key: tenant, operator: Exists}'
NEU='- {key: tenant-tier, operator: In, values: [standard]}'
sed -i.bak "s/$ALT/$NEU/" policies/no-world-egress.yaml
kyverno test tests/no-world-egress --remove-color | grep Summary   # that was the CI
kubectl apply -f policies/no-world-egress.yaml

Time limit: 10 min

Symptoms:

Test Summary: 4 tests passed and 0 tests failed
$ kubectl apply --dry-run=server -f tests/no-world-egress/netpols.yaml
networkpolicy.networking.k8s.io/api-to-db created (server dry run)
networkpolicy.networking.k8s.io/internet created (server dry run)
networkpolicy.networking.k8s.io/alles-raus created (server dry run)
...

Diagnosis path:

kyverno test tests/no-world-egress --remove-color \
  | awk -F'│' '/│ [0-9]/ {print $5, $6, $7}'
kyverno apply policies/no-world-egress.yaml -r tests/no-world-egress/netpols.yaml \
  -f tests/no-world-egress/values.yaml --remove-color | tail -1
kubectl get ns -l tenant-tier

Expected (the first five lines tested live, abridged):

 networking.k8s.io/v1/NetworkPolicy/tenant-a/api-to-db      Pass   Excluded
 networking.k8s.io/v1/NetworkPolicy/tenant-a/dns-ueberall   Pass   Excluded
 networking.k8s.io/v1/NetworkPolicy/tenant-a/internet       Pass   Excluded
 networking.k8s.io/v1/NetworkPolicy/tenant-a/alles-raus     Pass   Excluded
pass: 0, fail: 0, warn: 0, error: 0, skip: 0
No resources found

The trap from Day 11, Drill 3: Excluded counts as passed, even if you expect fail. The new selector matches nothing, neither in the test (values.yaml does not know tenant-tier) nor in the cluster (no namespace carries it).

Solution: Roll back, then sharpen the CI so that a test without any match turns red.

mv policies/no-world-egress.yaml.bak policies/no-world-egress.yaml
kubectl apply -f policies/no-world-egress.yaml
cat > hit-guard.sh <<'EOF'
#!/usr/bin/env bash
# hit-guard.sh - every test folder must really check at least one object
#
#   hit-guard.sh <test-ordner>...
#
# Exit 0 = every folder has a result with REASON "Ok", 1 = only Excluded/Skip.
set -uo pipefail
rc=0
for d in "$@"; do
  out=$(kyverno test "$d" --remove-color 2>&1)
  if grep -Eq '│ (Pass|Fail) +│ Ok ' <<<"$out"; then
    echo "✔ $d prüft echte Objekte"
  else
    echo "✘ $d: kein Objekt geprüft (nur Excluded?)"
    rc=1
  fi
done
exit $rc
EOF
chmod +x hit-guard.sh
./hit-guard.sh tests/*/

Expected (tested live):

✔ tests/no-world-egress/ prüft echte Objekte
✔ tests/require-default-deny/ prüft echte Objekte

With the broken version, the script reports ✘ tests/no-world-egress/: kein Objekt geprüft (nur Excluded?) (no object checked) and exit 1. It belongs as a second step after kyverno test in the pipeline from Day 11.

Lessons:

  • Green only means “expectation met”. A test that checks nothing is always green.
  • values.yaml is a copy of the cluster labels and silently goes stale. Generate it from the cluster (kubectl get ns -o json | jq …) or check it against the cluster.
  • For label changes, change the cluster first (label the namespaces), then switch the policies. Never the other way around.

Save your progress? Sign up for free

Drill 8: toFQDNs after a CDN change

Scenario: Last week a colleague “slimmed down” client-fqdn (Day 4): IPs from dig instead of domains, because “the DNS proxy is a single point of failure”. Today the CDN behind example.com changed its addresses.

This drill runs on cilium-lab. Build it as on Day 4, Drills 1–2, with the demo app (does not count toward the time). With 8 GB of RAM for Docker, both clusters run single-node side by side; otherwise run the checkpoint for kyverno-lab first, then delete it.

kubectl config use-context kind-cilium-lab
kubectl apply -f ~/guard-lab/code/cilium/client-fqdn.yaml
TIMEOUT=5 cc default/client https://example.com allow

Trigger:

cat > client-fqdn-pinned.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: client-fqdn            # same name: replaces the policy from Day 4
  namespace: default
spec:
  endpointSelector:
    matchLabels:
      app: client
  egress:
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
  - toCIDR:
    - 192.0.2.80/32            # "IP from last week" (documentation range, simulated)
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
EOF
kubectl apply -f client-fqdn-pinned.yaml

Time limit: 10 min

Symptoms:

✘ blockiert default/client -> https://example.com (Timeout)
✘ blockiert default/client -> https://api.github.com (Timeout)

Diagnosis path:

kubectl exec client -- nslookup example.com | grep -A1 '^Name'
hubble observe -P --from-pod default/client --verdict DROPPED --last 3
kubectl get cnp client-fqdn -o yaml | yq '.spec.egress[] | keys'

Expected (abridged, not tested live, may differ depending on version):

Name:   example.com
Address: 23.192.228.84
... default/client:40112 (ID:23412) -> 23.192.228.84:443 (world) Policy denied
  DROPPED (TCP Flags: SYN)
- toEndpoints
- toPorts
- toCIDR
- toPorts

DNS works, the connection to the current IP is dropped. The policy only knows one IP, and it is no longer correct.

Solution, round 1: back to domains. First a half fix, the way it happens under stress: toFQDNs back in, the DNS rule stays pure L4.

yq 'del(.spec.egress[0].toPorts[0].rules)' ~/guard-lab/code/cilium/client-fqdn.yaml \
  | kubectl apply -f -
TIMEOUT=5 cc default/client https://docs.github.com
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg fqdn cache list | grep -c docs.github

Expected (abridged, not tested live, may differ depending on version):

ciliumnetworkpolicy.cilium.io/client-fqdn configured
✘ blockiert default/client -> https://docs.github.com (Timeout)
0

docs.github.com matches *.github.com, but never lands in the FQDN cache: Without rules.dns, DNS bypasses the proxy and Cilium learns no IPs (Day 4, Drill 6). Names that are already known keep working until their cache entries expire. That makes the error treacherous.

Solution, round 2: the original file from Git.

kubectl apply -f ~/guard-lab/code/cilium/client-fqdn.yaml
TIMEOUT=5 cc default/client https://docs.github.com allow
TIMEOUT=5 cc default/client https://example.com allow
TIMEOUT=5 cc default/client https://kubernetes.io deny

Expected (abridged, not tested live, may differ depending on version):

ciliumnetworkpolicy.cilium.io/client-fqdn configured
✔ erlaubt   default/client -> https://docs.github.com (HTTP 200)
✔ erlaubt   default/client -> https://example.com (HTTP 200)
✘ blockiert default/client -> https://kubernetes.io (Timeout)

Lessons:

  • CDN and SaaS IPs change without notice. Use toCIDR only for networks that you or the partner own by contract.
  • toFQDNs always needs a DNS rule with rules.dns to kube-dns. A test with a new name proves it; a known name does not.
  • The DNS proxy really is in the data path. The answer to that is monitoring and capacity (Enterprise Note), not switching it off.

Save your progress? Sign up for free

Scoring

Full points only within the time limit and without uncovering the diagnosis path. Up to double the time or with peeking, you get half; after that, nothing.

Drill Time limit Points
1 Egress hardening without DNS 5 min 10
2 Kyverno gone, failurePolicy Fail 10 min 15
3 GeneratingPolicy spreads a mistake 10 min 15
4 Unsigned hotfix image 10 min 15
5 Label drift between tenants 10 min 10
6 Monitoring locked out 10 min 10
7 CI green, cluster red 10 min 15
8 toFQDNs after a CDN change 10 min 10
Total 75 min 100

With 70 points or more, you are ready for on-call duty. For every drill you missed, write a line in the runbook: symptom, first command, solution.

Checkpoint

After all drills, the end state must be clean, not just the symptom gone.

#!/usr/bin/env bash
cd ~/guard-lab/tag14 || exit 1
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
CC=~/guard-lab/code/demo/conncheck.sh; M=~/guard-lab/code/np-tests/np-matrix.sh
kubectl config use-context kind-kyverno-lab >/dev/null
CC=$CC $M wettkampf.tsv >/dev/null && ok "Wettkampf-Matrix grün" \
  || nok "Wettkampf-Matrix rot"
D=deploy/kyverno-admission-controller
img=$(kubectl -n kyverno get $D -o jsonpath='{.spec.template.spec.containers[0].image}')
[[ $img == *:v1.19.1 ]] && ok "Kyverno-Image v1.19.1" || nok "Kyverno-Image: $img"
r=$(kubectl -n kyverno get $D -o jsonpath='{.status.readyReplicas}')
[ "${r:-0}" -ge 3 ] && ok "Admission 3 Replikas" || nok "Admission: ${r:-0} Replikas"
kubectl -n shop run t -l app=t --image=nginxinc/nginx-unprivileged:latest \
  --dry-run=server >/dev/null 2>&1 \
  && nok ":latest kommt durch" || ok ":latest abgelehnt"
kubectl -n shop get netpol allow-dns >/dev/null 2>&1 \
  && ok "shop hat allow-dns" || nok "shop ohne allow-dns"
[ "$(kubectl get ns -l purpose=monitoring -o name)" = namespace/monitoring ] \
  && ok "purpose nur an monitoring" || nok "purpose an fremden Namespaces"
kubectl get ns monitoring -o jsonpath='{.metadata.labels.tenant}' | grep -q . \
  && nok "monitoring ist noch Mandant" || ok "monitoring ist kein Mandant"
k=$(kubectl get vpol no-world-egress \
  -o jsonpath='{.spec.matchConstraints.namespaceSelector.matchExpressions[0].key}')
[ "$k" = tenant ] && ok "no-world-egress trifft Mandanten" || nok "Selector: $k"
./hit-guard.sh tests/*/ >/dev/null && ok "hit-guard grün" || nok "hit-guard rot"
kubectl -n policy-exceptions get policyexceptions.policies.kyverno.io \
  hotfix-web-unsigned >/dev/null 2>&1 \
  && nok "Hotfix-Ausnahme noch da" || ok "Hotfix-Ausnahme entfernt"
if kubectl config get-contexts kind-cilium-lab >/dev/null 2>&1; then
  kubectl config use-context kind-cilium-lab >/dev/null
  for u in https://example.com https://docs.github.com; do
    TIMEOUT=5 $CC default/client "$u" allow >/dev/null \
      && ok "cilium: client -> $u" || nok "cilium: client -> $u"
  done
fi

Challenge

Prevent Drills 5 and 6 via admission: A ValidatingPolicy protect-ns-labels allows the label purpose only on the namespace monitoring, and every namespace with the prefix tenant- needs the label tenant. Test offline against four namespaces.

Hint: startsWith() and the in operator on the label map are enough; fetch the labels once via .?labels.orValue({}) into a variable.

Solution

cd ~/guard-lab/tag14
cat > protect-ns-labels.yaml <<'EOF'
# Day 14: protect platform labels on namespaces (prevent label drift)
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: protect-ns-labels
spec:
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE, UPDATE]
      resources: [namespaces]
  variables:
  - name: name
    expression: object.metadata.name
  - name: labels
    expression: object.metadata.?labels.orValue({})
  validations:
  - expression: >-
      !('purpose' in variables.labels) || variables.name == 'monitoring'
    messageExpression: >-
      'Namespace ' + variables.name + ': Label purpose ist reserviert (nur monitoring)'
  - expression: >-
      !variables.name.startsWith('tenant-') || 'tenant' in variables.labels
    messageExpression: >-
      'Namespace ' + variables.name + ': Mandanten-Namespaces brauchen das Label tenant'
EOF
cat > drift-ns.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-e
  labels: {guard-lab: demo, team: platform, purpose: monitoring}
---
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-f
  labels: {guard-lab: demo, team: tenant-f, tenent: f}
---
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-g
  labels: {guard-lab: demo, team: tenant-g, tenant: g}
---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels: {guard-lab: demo, team: platform, purpose: monitoring}
EOF
kyverno apply protect-ns-labels.yaml -r drift-ns.yaml --remove-color

Expected (tested live):

Applying 1 policy rule(s) to 4 resource(s)...
policy protect-ns-labels -> resource /Namespace/tenant-e failed:
1 -  Namespace tenant-e: Label purpose ist reserviert (nur monitoring)
policy protect-ns-labels -> resource /Namespace/tenant-f failed:
1 -  Namespace tenant-f: Mandanten-Namespaces brauchen das Label tenant

pass: 2, fail: 2, warn: 0, error: 0, skip: 0

tenant-e violates both rules; only the first is reported (Day 8, Drill 2). The typo tenent in tenant-f gets caught; monitoring and tenant-g pass. The policy does not prevent Drill 6: monitoring is not named tenant-…. For that you need a third rule, such as “never tenant on namespaces with purpose”.

Save your progress? Sign up for free

Enterprise Note

  • These drills belong in a quarterly game day, in staging with real alerts. What gets measured is the time to detection, not just the time to repair.
  • For every drill, a runbook with an emergency exit: delete webhooks, create an exception, roll back the policy. The permissions for that sit with a break-glass group, not with everyone.
  • Alerts that show these failures early: Hubble drops with policy-verdict:none, kyverno_admission_requests_total{request_allowed="false"}, up == 0 per target, missing Kyverno endpoints, growing DNS proxy latency.
  • Trigger labels (tenant, purpose) and GeneratingPolicies go through change review like code. Before: kyverno apply against real namespaces; after: the target matrix.
  • Every incident ends with a new test: a matrix line, a kyverno test case or a Chainsaw step. Otherwise the same mistake comes back.