Day 14: Competition – Disaster Drills
Goal: Find and fix eight typical network and admission failures under time pressure without tearing open new holes. · Duration: approx. 180 min · Prerequisite: Day 13 (Drill 8 additionally Day 4)
Briefing
- Every drill has a trigger. Fire it, then start the stopwatch and only stop it once the competition matrix is green again.
- Measure first, then change:
np-matrix.sh,conncheck.sh, events, reports, logs. Whoever guesses loses time. - Read only the scenario and symptoms. Uncover the diagnosis path and solution only when you are stuck. That costs half the points.
- Drills 1–7 run on
kyverno-lab, Drill 8 oncilium-lab. - None of the cluster-related outputs have been tested live. Checked live are the Kyverno CLI 1.19.1 (Drills 3, 7, challenge) and the syntax of all files.
failure ──► symptom (matrix ✘, event, log) ──► diagnosis ──► smallest fix
└──► lasting lesson (Git, policy)
Drill 0: Lineup (untimed)
A defined starting state: registry and lab CA from Day 10, Kyverno HA
from Day 12, policies from Days 8 to 13. Everything comes fresh from
~/guard-lab/code.
mkdir -p ~/guard-lab/tag14 && cd ~/guard-lab/tag14
kubectl config use-context kind-kyverno-lab
alias cc=~/guard-lab/code/demo/conncheck.sh
cp -R ~/guard-lab/code/kyverno-ops/tag13/{policies,tests} .
(cd ~/guard-lab/tag10 && ./setup-registry.sh)
ARCH=$(kubectl get node -o jsonpath='{.items[0].status.nodeInfo.architecture}')
crane copy --insecure --platform linux/$ARCH \
nginxinc/nginx-unprivileged:1.31-alpine localhost:5001/shop/web:1.31 2>/dev/null
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.9.1 \
-f ~/guard-lab/tag12/values-ha.yaml \
--set-file global.caCertificates.data=$HOME/guard-lab/tag10/certs/registry.crt \
--wait --timeout 5m
kubectl -n kyverno rollout restart deploy kyverno-admission-controller \
kyverno-background-controller kyverno-reports-controller
kubectl -n kyverno rollout status deploy kyverno-admission-controller --timeout=180sExpected (abridged, not tested live, may differ depending on version):
✔ kyverno-lab-control-plane vertraut kind-registry:5000
✔ Registry erreichbar: https://localhost:5001
Release "kyverno" has been upgraded. Happy Helming!
deployment "kyverno-admission-controller" successfully rolled out
Why the CA again? values-ha.yaml from Day 12 does not
contain it. Every helm upgrade -f values-ha.yaml without
--set-file silently throws it out, and Day 10 ends with
x509: certificate signed by unknown authority.
Now the policies and the template for monitoring:
K=~/guard-lab/code/kyverno
kubectl apply -f $K/tag08/image-hygiene.yaml \
-f ~/guard-lab/tag10/verify-shop-images.yaml \
-f ~/guard-lab/tag11/policy-repo/policies/exception-governance.yaml \
-f ~/guard-lab/tag12/no-privileged-vap.yaml \
-f $K/tag09/tenant-netpol.yaml -f policies/
kubectl patch gpol tenant-netpol --type merge \
-p '{"spec":{"evaluation":{"generateExisting":{"enabled":true}}}}'
kubectl create namespace netpol-templates
kubectl apply -f $K/tag09/tpl-allow-monitoring.yaml -f $K/tag09/clone-monitoring.yaml
sleep 20
kubectl get netpol -n tenant-aExpected (abridged, not tested live, may differ depending on version):
NAME POD-SELECTOR AGE
allow-dns <none> 15s
allow-monitoring tier=api 12s
allow-same-namespace <none> 2d
default-deny <none> 15s
The target matrix for the entire competition, checked with
np-matrix.sh from Day 6:
cat > wettkampf.tsv <<'EOF'
# von nach soll
default/client http://frontend.shop allow
shop/frontend http://api.shop allow
shop/api tcp://db.shop:5432 allow
tenant-a/api http://api.tenant-a allow
tenant-a/api http://api.tenant-b deny
tenant-b/api http://api.tenant-a deny
default/client http://api.tenant-a deny
monitoring/scraper http://api.tenant-a:9113/metrics allow
monitoring/scraper http://api.tenant-b:9113/metrics allow
monitoring/scraper http://api.tenant-a deny
EOF
M=~/guard-lab/code/np-tests/np-matrix.sh
$M wettkampf.tsv | tail -1Expected (abridged, not tested live, may differ depending on version):
10 Tests: 10 ✔, 0 ✘
Not green? Then the starting state is wrong. Fix it first, then start.
Warning: Work only in the lab. Several triggers are deliberately brutal (deleting webhooks, labels on platform namespaces). In production they belong in a game day protocol with approval.
Save your progress? Sign up for free
Drill 1: Egress hardening without DNS
Scenario: The shop team wants to
“prevent data exfiltration” and only allows egress within the namespace.
Two minutes later, support reports: orders are hanging.
Trigger:
cat > egress-lockdown.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: egress-lockdown
namespace: shop
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- podSelector: {} # "only within our own namespace"
EOF
kubectl apply -f egress-lockdown.yaml; sleep 15Time limit: 5 min
Symptoms:
$ kubectl exec client -- curl -s -o /dev/null -w '%{http_code}\n' http://frontend.shop/
200
$ kubectl exec client -- curl -s -o /dev/null -w '%{http_code}\n' \
http://frontend.shop/api/orders
502
Diagnosis path:
$M wettkampf.tsv | grep '✘'
kubectl -n shop logs deploy/frontend --tail=3 | grep -o 'could not be resolved.*'
kubectl -n shop exec deploy/frontend -- nslookup api.shop.svc.cluster.local
kubectl -n shop get netpol
kubectl -n shop describe netpol egress-lockdown | sed -n '/Spec/,$p'Expected (abridged, not tested live, may differ depending on version):
shop/frontend http://api.shop allow ✘ deny
could not be resolved (110: Operation timed out)
;; connection timed out; no servers could be reached
NAME POD-SELECTOR AGE
egress-lockdown <none> 1m
Allowing egress traffic:
To Port: <any> (traffic allowed to all ports)
To:
PodSelector: <none>
The frontend is alive, only /api/ fails: nginx
re-resolves api.shop every 10 s. CoreDNS lives in
kube-system, and egress there is now closed (Day 2, Drill
1).
Solution:
kubectl apply -f ~/guard-lab/tag02/allow-dns.yaml
sleep 5
$M wettkampf.tsv | tail -1Expected (abridged, not tested live):
networkpolicy.networking.k8s.io/allow-dns created
10 Tests: 10 ✔, 0 ✘
The hardening stays, DNS is added on top. Do not delete the policy; the team wanted it.
Lessons:
- Egress default deny and
allow-dnsare a pair. Never roll out one without the other. - “Page loads, feature broken” is the typical picture: ingress ok, egress DNS gone.
- Here
conncheck.shreports “Timeout”, not “DNS-Fehler” (DNS error): The query starves.
Save your progress? Sign up for free
Drill 2: Kyverno gone, failurePolicy Fail
Scenario: A colleague tests a Kyverno update by hand, “real quick”. Afterwards no deployment goes through anymore, and scaling back up does not help.
Trigger:
kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=0
kubectl -n kyverno set image deploy/kyverno-admission-controller \
kyverno=reg.kyverno.io/kyverno/kyverno:v1.19.99Time limit: 10 min
Symptoms:
$ kubectl -n shop rollout restart deploy frontend
error: failed to patch: Internal error occurred: failed calling webhook
"…validate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc
.kyverno.svc:443/…": no endpoints available for service "kyverno-svc"
Diagnosis path:
kubectl -n kyverno get deploy kyverno-admission-controller
kubectl -n kyverno get endpointslice -l kubernetes.io/service-name=kyverno-svc
kubectl get validatingwebhookconfigurations -l webhook.kyverno.io/managed-by=kyverno \
-o custom-columns=NAME:.metadata.name,POLICY:.webhooks[*].failurePolicy
kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=3
sleep 20
kubectl -n kyverno get pods -l app.kubernetes.io/component=admission-controllerExpected (abridged, not tested live, may differ depending on version):
NAME READY UP-TO-DATE AVAILABLE AGE
kyverno-admission-controller 0/0 0 0 3d
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kyverno-svc-xxxxx IPv4 <unset> <unset> 3d
NAME POLICY
kyverno-resource-validating-webhook-cfg Fail,Ignore,Fail
...
kyverno-admission-controller-7f9c…-4kq2p 0/1 ImagePullBackOff 0 20s
Webhooks registered, no endpoint, the new image does not exist. Scaling alone heals nothing.
Solution: First the emergency exit, then the cause, then the proof.
kubectl -n shop run priv --image=nginxinc/nginx-unprivileged:1.31-alpine \
--privileged --dry-run=server -o name 2>&1 | tail -1
kubectl delete validatingwebhookconfiguration,mutatingwebhookconfiguration \
-l webhook.kyverno.io/managed-by=kyverno
kubectl -n shop rollout restart deploy frontend
kubectl -n kyverno rollout undo deploy kyverno-admission-controller
kubectl -n kyverno rollout status deploy kyverno-admission-controller --timeout=180s
sleep 15
kubectl get validatingwebhookconfigurations -l webhook.kyverno.io/managed-by=kyverno \
--no-headers | wc -l
kubectl -n shop run x -l app=x --image=nginxinc/nginx-unprivileged:latest \
--dry-run=server -o name 2>&1 | sed 's/.*denied the request: //'Expected (abridged, not tested live, may differ depending on version):
The pods "priv" is invalid: : ValidatingAdmissionPolicy 'no-privileged' with binding
'no-privileged-binding' denied request: Privilegierte Container sind verboten.
validatingwebhookconfiguration.admissionregistration.k8s.io "kyverno-…" deleted
...
deployment.apps/frontend restarted
deployment.apps/kyverno-admission-controller rolled back
deployment "kyverno-admission-controller" successfully rolled out
4
Policy image-hygiene failed: Fester Tag oder Digest nötig, nicht :latest:
nginxinc/nginx-unprivileged:latest
The first line is the good news: The native VAP from Day 12 works
even without Kyverno. rollout undo brings back the old
image; the replicas are already at 3. Kyverno recreates its webhooks
itself on startup.
Lessons:
- Deleting webhooks is the emergency exit, not a fix. Until Kyverno runs again, only what lives in the API server as VAP or PSA applies. Record the time window in the incident log.
- Never change Kyverno via
kubectl set image. Versions only via Helm and Git (Day 12). kube-systemandkyvernoare excluded. Otherwise the cluster locks itself out on restart, and the emergency exit is the only way.
Save your progress? Sign up for free
Drill 3: A GeneratingPolicy spreads a mistake
Scenario: The platform “simplifies”
tenant-onboarding: allow-same-namespace should
only govern ingress. Thanks to synchronize, the rollout
reaches every tenant within seconds.
Trigger:
sed 's/"policyTypes": dyn(\["Ingress", "Egress"\]),/"policyTypes": dyn(["Ingress"]),/' \
policies/tenant-onboarding.yaml > tenant-onboarding-v2.yaml
kubectl apply -f tenant-onboarding-v2.yaml; sleep 20Time limit: 10 min
Symptoms: Traffic within all tenants breaks down; between tenants everything stays sealed. A manual repair only lasts seconds.
tenant-a/api http://api.tenant-a allow ✘ deny
10 Tests: 9 ✔, 1 ✘
Diagnosis path:
kubectl -n tenant-a describe netpol allow-same-namespace | grep -E 'egress|Types'
kubectl get netpol -A -l generate.kyverno.io/policy-name=tenant-onboarding \
-o custom-columns=NS:.metadata.namespace,TYPES:.spec.policyTypes
diff policies/tenant-onboarding.yaml tenant-onboarding-v2.yaml
kyverno apply tenant-onboarding-v2.yaml \
-r ~/guard-lab/code/kyverno-ops/tag13/tenant-d.yaml --remove-color \
| sed -n '/allow-same-namespace/,/^---/p' | grep -A1 policyTypesExpected (diff and kyverno apply tested live, rest
abridged):
Not affecting egress traffic
Policy Types: Ingress
NS TYPES
tenant-a ["Ingress"]
tenant-b ["Ingress"]
tenant-d ["Ingress"]
36c36
< "policyTypes": dyn(["Ingress", "Egress"]),
---
> "policyTypes": dyn(["Ingress"]),
policyTypes:
- Ingress
The egress rule is still in the object, but without
Egress in policyTypes it is ignored.
default-deny thus also blocks traffic within the
namespace.
Solution: Fix the source, not the copies.
kubectl -n tenant-a patch netpol allow-same-namespace --type merge \
-p '{"spec":{"policyTypes":["Ingress","Egress"]}}'; sleep 10
kubectl -n tenant-a get netpol allow-same-namespace -o jsonpath='{.spec.policyTypes}'
echo
kubectl apply -f policies/tenant-onboarding.yaml; sleep 20
$M wettkampf.tsv | tail -1Expected (abridged, not tested live, may differ depending on version):
networkpolicy.networking.k8s.io/allow-same-namespace patched
["Ingress"]
generatingpolicy.policies.kyverno.io/tenant-onboarding configured
10 Tests: 10 ✔, 0 ✘
The hotfix in the namespace is reverted (Day 9, Drill 4). Only the corrected policy spreads everywhere again.
Lessons:
synchronizespreads mistakes just as fast as corrections. Generated policies need the same tests as handwritten ones:kyverno applyagainst a sample namespace shows the result before the merge.- During an incident, always fix the policy in Git. Hand patches to generated objects are lost minutes.
- A matrix case “allowed within the tenant” belongs in every target matrix, not just the denials.
Save your progress? Sign up for free
Drill 4: Unsigned hotfix image during an incident
Scenario: A production bug in the shop. The hotfix
is built, but the CI signing step has failed.
verify-shop-images (Day 10) does not let the image through.
The incident manager wants to see the hotfix live in 10 minutes.
Trigger:
crane copy --insecure localhost:5001/shop/web:1.31 localhost:5001/shop/hotfix:1.31 \
2>/dev/null
kubectl -n shop create deploy hotfix --image=kind-registry:5000/shop/hotfix:1.31 \
--dry-run=client -o yaml \
| kubectl set resources --local -f - --requests=cpu=10m,memory=16Mi \
--limits=memory=64Mi -o yaml > hotfix.yaml
kubectl apply -f hotfix.yamlTime limit: 10 min
Symptoms:
Error from server: error when creating "hotfix.yaml": admission webhook "…" denied the
request: Policy verify-shop-images failed: Image ist nicht mit dem Lab-Key signiert.
Diagnosis path:
kubectl -n kyverno logs deploy/kyverno-admission-controller --since=5m \
| grep -oE 'no signatures found|x509: [a-z ]+' | sort | uniq -c
crane ls --insecure localhost:5001/shop/hotfix
kubectl get ivpol verify-shop-images -o jsonpath='{.spec.matchImageReferences}'
echoExpected (abridged, not tested live, may differ depending on version):
1 no signatures found
1.31
[{"glob":"kind-registry:5000/shop/*"}]
no signatures found means: The policy is correct, the
image really is unsigned. If it says
x509: certificate signed by unknown authority, Kyverno is
missing the lab CA (Drill 0), and every image fails.
That would be a different incident.
Solution: An exception, as narrow and as short as possible, with a ticket. Afterwards, sign the image retroactively and remove the exception immediately.
BIS=$(date -u -v+24H +%Y-%m-%dT%H:%M:%SZ) # Linux: date -u -d '+24 hours' +%FT%TZ
cat > hotfix-exception.yaml <<EOF
# Day 14: emergency exception - unsigned hotfix image, 24 h, ticket mandatory
apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata:
name: hotfix-web-unsigned
namespace: policy-exceptions
spec:
policyRefs:
- name: verify-shop-images
kind: ImageValidatingPolicy
matchConditions:
- name: nur-shop-hotfix
expression: >-
object.metadata.namespace == 'shop' &&
object.metadata.?labels[?'app'].orValue('') == 'hotfix'
expiresAt: "$BIS"
properties:
ticket: INC-0815
owner: team-shop
reason: Hotfix ohne Signatur, CI-Signierschritt ausgefallen
EOF
kyverno apply ~/guard-lab/tag11/policy-repo/policies/exception-governance.yaml \
-r hotfix-exception.yaml --remove-color | tail -1
kubectl apply -f hotfix-exception.yaml
kubectl apply -f hotfix.yaml
kubectl -n shop rollout status deploy hotfix --timeout=90sExpected (first line tested live, rest abridged, not tested live):
pass: 1, fail: 0, warn: 0, error: 0, skip: 0
policyexception.policies.kyverno.io/hotfix-web-unsigned created
deployment.apps/hotfix created
deployment "hotfix" successfully rolled out
After the incident, you add the signature retroactively (key and
offline.json from Day 10) and prove that it works without
the exception:
cd ~/guard-lab/tag10
D=$(crane digest --insecure localhost:5001/shop/hotfix:1.31)
cosign sign --registry-cacert certs/registry.crt --key cosign.key \
--signing-config offline.json -y "localhost:5001/shop/hotfix@$D"
cosign attest --registry-cacert certs/registry.crt --key cosign.key \
--signing-config offline.json -y --type spdxjson --predicate sbom.spdx.json \
"localhost:5001/shop/hotfix@$D"
cd ~/guard-lab/tag14
kubectl -n policy-exceptions delete policyexceptions.policies.kyverno.io \
hotfix-web-unsigned
kubectl -n shop rollout restart deploy hotfix
kubectl -n shop rollout status deploy hotfix --timeout=90sExpected (abridged, not tested live, may differ depending on version):
policyexception.policies.kyverno.io "hotfix-web-unsigned" deleted
deployment.apps/hotfix restarted
deployment "hotfix" successfully rolled out
Why the SBOM too? verify-shop-images checks the
signature and the SPDX attestation (Day 10). The SBOM
from Day 10 fits, because the digest is the same.
Lessons:
- During an incident, an exception is the right tool; switching off
the policy never is.
exception-governance(Day 11) enforces a ticket, an owner and an expiration. - Cut it narrow: one policy, one namespace, one label.
expiresAtin hours, not in months.delete-expired-exceptionscleans up the rest (Day 12). - Check the Kyverno logs first: missing signature, wrong key, missing CA or registry gone are four different incidents with the same message to the client.
Save your progress? Sign up for free
Drill 5: Label drift opens the tenant boundary
Scenario: A team creates tenant-e and
copies the manifest of the monitoring namespace for it.
Nobody notices anything until a pentest fetches other tenants’ metrics
from tenant-e.
Trigger:
cat > tenant-e.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: tenant-e
labels: {guard-lab: demo, team: platform, purpose: monitoring} # copied
EOF
kubectl apply -f tenant-e.yaml
sed 's/namespace: default/namespace: tenant-e/' ~/guard-lab/code/demo/40-client.yaml \
| kubectl apply -f -
kubectl -n tenant-e wait --for=condition=Ready pod/client --timeout=60sTime limit: 10 min
Symptoms:
$ cc tenant-e/client http://api.tenant-a:9113/metrics deny
✔ erlaubt tenant-e/client -> http://api.tenant-a:9113/metrics (HTTP 200)
<-- erwartet: deny
Diagnosis path:
kubectl get ns tenant-e --show-labels
kubectl get netpol -n tenant-e
kubectl -n tenant-a describe netpol allow-monitoring | sed -n '/Allowing/,/Not/p'
kubectl get ns -l purpose=monitoringExpected (abridged, not tested live, may differ depending on version):
NAME STATUS AGE LABELS
tenant-e Active 2m guard-lab=demo,kubernetes.io/metadata.name=tenant-e,…
No resources found in tenant-e namespace.
Allowing ingress traffic:
To Port: 9113/TCP
From:
NamespaceSelector: purpose=monitoring
NAME STATUS AGE
monitoring Active 5d
tenant-e Active 2m
Two mistakes interlock: Without the label tenant,
tenant-e has no default deny (egress open). And with
purpose=monitoring, it counts as monitoring in every tenant
(Day 3, Drill 6: labels as a contract).
Solution: First get the labels right, then harden the template so that a wrong label alone is never enough again.
kubectl label ns tenant-e purpose- team=tenant-e tenant=e --overwrite
kubectl -n netpol-templates patch netpol allow-monitoring --type json -p '[{"op":
"replace","path":"/spec/ingress/0/from","value":[{"namespaceSelector":{
"matchLabels":{"kubernetes.io/metadata.name":"monitoring"}},
"podSelector":{"matchLabels":{"app":"scraper"}}}]}]'
sleep 20
kubectl get netpol -n tenant-e
cc tenant-e/client http://api.tenant-a:9113/metrics deny
$M wettkampf.tsv | tail -1Expected (abridged, not tested live, may differ depending on version):
namespace/tenant-e labeled
networkpolicy.networking.k8s.io/allow-monitoring patched
NAME POD-SELECTOR AGE
allow-dns <none> 15s
allow-monitoring tier=api 15s
allow-same-namespace <none> 15s
default-deny <none> 15s
✘ blockiert tenant-e/client -> http://api.tenant-a:9113/metrics (Timeout)
10 Tests: 10 ✔, 0 ✘
The label tenant triggers generation; the changed
template synchronizes into all tenants (Day 9, Drill 6). The new source
is an AND of kubernetes.io/metadata.name and
app=scraper, which no team can forge.
Lessons:
- Copied manifests copy permissions along with them. Self-assigned
labels like
purposeare only as secure as the rule that protects them (challenge). - Always select foreign namespaces via
kubernetes.io/metadata.name(Day 3). - A missing default deny is not an error that any policy reports.
require-default-deny(Day 13) only checks namespaces withtenant. If the label is missing, the check is missing too.
Save your progress? Sign up for free
Drill 6: A rollout locks out monitoring
Scenario: The platform wants quota and Pod Security
“for monitoring too” and attaches the tenant label to the namespace.
Shortly afterwards, every alert fires TargetDown.
Trigger:
kubectl label ns monitoring tenant=platform
sleep 20Time limit: 10 min
Symptoms:
$ kubectl -n monitoring logs deploy/scraper --tail=3
FAIL shop
FAIL tenant-a
FAIL tenant-b
Diagnosis path:
$M wettkampf.tsv | grep '✘'
kubectl get netpol,resourcequota,rolebinding -n monitoring \
-L generate.kyverno.io/policy-name
kubectl get ns monitoring -o json | jq -c .metadata.labelsExpected (abridged, not tested live, may differ depending on version):
monitoring/scraper http://api.tenant-a:9113/metrics allow ✘ deny
monitoring/scraper http://api.tenant-b:9113/metrics allow ✘ deny
NAME … POLICY-NAME
networkpolicy…/allow-dns … tenant-netpol
networkpolicy…/allow-monitoring … clone-monitoring
networkpolicy…/allow-same-namespace … tenant-onboarding
networkpolicy…/default-deny … tenant-netpol
resourcequota/tenant-quota … tenant-onboarding
rolebinding…/team-edit … tenant-onboarding
{"guard-lab":"demo",…,"pod-security.kubernetes.io/enforce":"restricted",…,
"tenant":"platform"}
One label, six objects and three PSA labels.
default-deny blocks the scraper’s egress, and
team-edit casually gives the group platform
edit in the namespace.
Solution: Removing the label is not enough. Generated objects and mutations stay in place (Day 9, Drill 7); you clean them up selectively:
kubectl label ns monitoring tenant- pod-security.kubernetes.io/enforce- \
pod-security.kubernetes.io/enforce-version- pod-security.kubernetes.io/warn-
P=tenant-netpol,tenant-onboarding,clone-monitoring
kubectl -n monitoring delete netpol,resourcequota,rolebinding \
-l "generate.kyverno.io/policy-name in ($P)"
sleep 15
kubectl -n monitoring logs deploy/scraper --tail=3
$M wettkampf.tsv | tail -1Expected (abridged, not tested live, may differ depending on version):
namespace/monitoring unlabeled
networkpolicy.networking.k8s.io "allow-dns" deleted from monitoring namespace
...
rolebinding.rbac.authorization.k8s.io "team-edit" deleted from monitoring namespace
ok shop
ok tenant-a
ok tenant-b
10 Tests: 10 ✔, 0 ✘
If the objects come back after deletion, the namespace still matches a GeneratingPolicy: check the labels again.
Lessons:
- A trigger label is a rollout switch for everything that hangs off
it: network, quota, RBAC, PSA. Before setting it, run
kyverno applyagainst the namespace (Day 13, Drill 5) and read the list. - Platform namespaces are not tenants. If monitoring wants default deny, it gets its own policies with egress to all scrape targets.
- The scraper log is the fastest sensor in the lab; in production,
up == 0per target.
Save your progress? Sign up for free
Drill 7: CI green, cluster red
Scenario: A pull request switches
no-world-egress to the new label schema
tenant-tier. kyverno test is green, review
done, merge, rollout. A week later the audit finds
0.0.0.0/0 in three tenants.
Trigger:
ALT='- {key: tenant, operator: Exists}'
NEU='- {key: tenant-tier, operator: In, values: [standard]}'
sed -i.bak "s/$ALT/$NEU/" policies/no-world-egress.yaml
kyverno test tests/no-world-egress --remove-color | grep Summary # that was the CI
kubectl apply -f policies/no-world-egress.yamlTime limit: 10 min
Symptoms:
Test Summary: 4 tests passed and 0 tests failed
$ kubectl apply --dry-run=server -f tests/no-world-egress/netpols.yaml
networkpolicy.networking.k8s.io/api-to-db created (server dry run)
networkpolicy.networking.k8s.io/internet created (server dry run)
networkpolicy.networking.k8s.io/alles-raus created (server dry run)
...
Diagnosis path:
kyverno test tests/no-world-egress --remove-color \
| awk -F'│' '/│ [0-9]/ {print $5, $6, $7}'
kyverno apply policies/no-world-egress.yaml -r tests/no-world-egress/netpols.yaml \
-f tests/no-world-egress/values.yaml --remove-color | tail -1
kubectl get ns -l tenant-tierExpected (the first five lines tested live, abridged):
networking.k8s.io/v1/NetworkPolicy/tenant-a/api-to-db Pass Excluded
networking.k8s.io/v1/NetworkPolicy/tenant-a/dns-ueberall Pass Excluded
networking.k8s.io/v1/NetworkPolicy/tenant-a/internet Pass Excluded
networking.k8s.io/v1/NetworkPolicy/tenant-a/alles-raus Pass Excluded
pass: 0, fail: 0, warn: 0, error: 0, skip: 0
No resources found
The trap from Day 11, Drill 3: Excluded counts as
passed, even if you expect fail. The new selector matches
nothing, neither in the test (values.yaml does not know
tenant-tier) nor in the cluster (no namespace carries
it).
Solution: Roll back, then sharpen the CI so that a test without any match turns red.
mv policies/no-world-egress.yaml.bak policies/no-world-egress.yaml
kubectl apply -f policies/no-world-egress.yaml
cat > hit-guard.sh <<'EOF'
#!/usr/bin/env bash
# hit-guard.sh - every test folder must really check at least one object
#
# hit-guard.sh <test-ordner>...
#
# Exit 0 = every folder has a result with REASON "Ok", 1 = only Excluded/Skip.
set -uo pipefail
rc=0
for d in "$@"; do
out=$(kyverno test "$d" --remove-color 2>&1)
if grep -Eq '│ (Pass|Fail) +│ Ok ' <<<"$out"; then
echo "✔ $d prüft echte Objekte"
else
echo "✘ $d: kein Objekt geprüft (nur Excluded?)"
rc=1
fi
done
exit $rc
EOF
chmod +x hit-guard.sh
./hit-guard.sh tests/*/Expected (tested live):
✔ tests/no-world-egress/ prüft echte Objekte
✔ tests/require-default-deny/ prüft echte Objekte
With the broken version, the script reports
✘ tests/no-world-egress/: kein Objekt geprüft (nur Excluded?)
(no object checked) and exit 1. It belongs as a second step after
kyverno test in the pipeline from Day 11.
Lessons:
- Green only means “expectation met”. A test that checks nothing is always green.
values.yamlis a copy of the cluster labels and silently goes stale. Generate it from the cluster (kubectl get ns -o json | jq …) or check it against the cluster.- For label changes, change the cluster first (label the namespaces), then switch the policies. Never the other way around.
Save your progress? Sign up for free
Drill 8: toFQDNs after a CDN change
Scenario: Last week a colleague “slimmed down”
client-fqdn (Day 4): IPs from dig instead of
domains, because “the DNS proxy is a single point of failure”. Today the
CDN behind example.com changed its addresses.
This drill runs on cilium-lab. Build it as on Day 4,
Drills 1–2, with the demo app (does not count toward the time). With 8
GB of RAM for Docker, both clusters run single-node side by side;
otherwise run the checkpoint for kyverno-lab first, then
delete it.
kubectl config use-context kind-cilium-lab
kubectl apply -f ~/guard-lab/code/cilium/client-fqdn.yaml
TIMEOUT=5 cc default/client https://example.com allowTrigger:
cat > client-fqdn-pinned.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: client-fqdn # same name: replaces the policy from Day 4
namespace: default
spec:
endpointSelector:
matchLabels:
app: client
egress:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
- toCIDR:
- 192.0.2.80/32 # "IP from last week" (documentation range, simulated)
toPorts:
- ports:
- port: "443"
protocol: TCP
EOF
kubectl apply -f client-fqdn-pinned.yamlTime limit: 10 min
Symptoms:
✘ blockiert default/client -> https://example.com (Timeout)
✘ blockiert default/client -> https://api.github.com (Timeout)
Diagnosis path:
kubectl exec client -- nslookup example.com | grep -A1 '^Name'
hubble observe -P --from-pod default/client --verdict DROPPED --last 3
kubectl get cnp client-fqdn -o yaml | yq '.spec.egress[] | keys'Expected (abridged, not tested live, may differ depending on version):
Name: example.com
Address: 23.192.228.84
... default/client:40112 (ID:23412) -> 23.192.228.84:443 (world) Policy denied
DROPPED (TCP Flags: SYN)
- toEndpoints
- toPorts
- toCIDR
- toPorts
DNS works, the connection to the current IP is dropped. The policy only knows one IP, and it is no longer correct.
Solution, round 1: back to domains. First a half
fix, the way it happens under stress: toFQDNs back in, the
DNS rule stays pure L4.
yq 'del(.spec.egress[0].toPorts[0].rules)' ~/guard-lab/code/cilium/client-fqdn.yaml \
| kubectl apply -f -
TIMEOUT=5 cc default/client https://docs.github.com
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg fqdn cache list | grep -c docs.githubExpected (abridged, not tested live, may differ depending on version):
ciliumnetworkpolicy.cilium.io/client-fqdn configured
✘ blockiert default/client -> https://docs.github.com (Timeout)
0
docs.github.com matches *.github.com, but
never lands in the FQDN cache: Without rules.dns, DNS
bypasses the proxy and Cilium learns no IPs (Day 4, Drill 6). Names that
are already known keep working until their cache entries expire. That
makes the error treacherous.
Solution, round 2: the original file from Git.
kubectl apply -f ~/guard-lab/code/cilium/client-fqdn.yaml
TIMEOUT=5 cc default/client https://docs.github.com allow
TIMEOUT=5 cc default/client https://example.com allow
TIMEOUT=5 cc default/client https://kubernetes.io denyExpected (abridged, not tested live, may differ depending on version):
ciliumnetworkpolicy.cilium.io/client-fqdn configured
✔ erlaubt default/client -> https://docs.github.com (HTTP 200)
✔ erlaubt default/client -> https://example.com (HTTP 200)
✘ blockiert default/client -> https://kubernetes.io (Timeout)
Lessons:
- CDN and SaaS IPs change without notice. Use
toCIDRonly for networks that you or the partner own by contract. toFQDNsalways needs a DNS rule withrules.dnsto kube-dns. A test with a new name proves it; a known name does not.- The DNS proxy really is in the data path. The answer to that is monitoring and capacity (Enterprise Note), not switching it off.
Save your progress? Sign up for free
Scoring
Full points only within the time limit and without uncovering the diagnosis path. Up to double the time or with peeking, you get half; after that, nothing.
| Drill | Time limit | Points |
|---|---|---|
| 1 Egress hardening without DNS | 5 min | 10 |
| 2 Kyverno gone, failurePolicy Fail | 10 min | 15 |
| 3 GeneratingPolicy spreads a mistake | 10 min | 15 |
| 4 Unsigned hotfix image | 10 min | 15 |
| 5 Label drift between tenants | 10 min | 10 |
| 6 Monitoring locked out | 10 min | 10 |
| 7 CI green, cluster red | 10 min | 15 |
| 8 toFQDNs after a CDN change | 10 min | 10 |
| Total | 75 min | 100 |
With 70 points or more, you are ready for on-call duty. For every drill you missed, write a line in the runbook: symptom, first command, solution.
Checkpoint
After all drills, the end state must be clean, not just the symptom gone.
#!/usr/bin/env bash
cd ~/guard-lab/tag14 || exit 1
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
CC=~/guard-lab/code/demo/conncheck.sh; M=~/guard-lab/code/np-tests/np-matrix.sh
kubectl config use-context kind-kyverno-lab >/dev/null
CC=$CC $M wettkampf.tsv >/dev/null && ok "Wettkampf-Matrix grün" \
|| nok "Wettkampf-Matrix rot"
D=deploy/kyverno-admission-controller
img=$(kubectl -n kyverno get $D -o jsonpath='{.spec.template.spec.containers[0].image}')
[[ $img == *:v1.19.1 ]] && ok "Kyverno-Image v1.19.1" || nok "Kyverno-Image: $img"
r=$(kubectl -n kyverno get $D -o jsonpath='{.status.readyReplicas}')
[ "${r:-0}" -ge 3 ] && ok "Admission 3 Replikas" || nok "Admission: ${r:-0} Replikas"
kubectl -n shop run t -l app=t --image=nginxinc/nginx-unprivileged:latest \
--dry-run=server >/dev/null 2>&1 \
&& nok ":latest kommt durch" || ok ":latest abgelehnt"
kubectl -n shop get netpol allow-dns >/dev/null 2>&1 \
&& ok "shop hat allow-dns" || nok "shop ohne allow-dns"
[ "$(kubectl get ns -l purpose=monitoring -o name)" = namespace/monitoring ] \
&& ok "purpose nur an monitoring" || nok "purpose an fremden Namespaces"
kubectl get ns monitoring -o jsonpath='{.metadata.labels.tenant}' | grep -q . \
&& nok "monitoring ist noch Mandant" || ok "monitoring ist kein Mandant"
k=$(kubectl get vpol no-world-egress \
-o jsonpath='{.spec.matchConstraints.namespaceSelector.matchExpressions[0].key}')
[ "$k" = tenant ] && ok "no-world-egress trifft Mandanten" || nok "Selector: $k"
./hit-guard.sh tests/*/ >/dev/null && ok "hit-guard grün" || nok "hit-guard rot"
kubectl -n policy-exceptions get policyexceptions.policies.kyverno.io \
hotfix-web-unsigned >/dev/null 2>&1 \
&& nok "Hotfix-Ausnahme noch da" || ok "Hotfix-Ausnahme entfernt"
if kubectl config get-contexts kind-cilium-lab >/dev/null 2>&1; then
kubectl config use-context kind-cilium-lab >/dev/null
for u in https://example.com https://docs.github.com; do
TIMEOUT=5 $CC default/client "$u" allow >/dev/null \
&& ok "cilium: client -> $u" || nok "cilium: client -> $u"
done
fiChallenge
Prevent Drills 5 and 6 via admission: A ValidatingPolicy
protect-ns-labels allows the label purpose
only on the namespace monitoring, and every namespace with
the prefix tenant- needs the label tenant.
Test offline against four namespaces.
Hint: startsWith() and the
in operator on the label map are enough; fetch the labels
once via .?labels.orValue({}) into a variable.
Solution
cd ~/guard-lab/tag14
cat > protect-ns-labels.yaml <<'EOF'
# Day 14: protect platform labels on namespaces (prevent label drift)
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
name: protect-ns-labels
spec:
validationActions: [Deny]
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [namespaces]
variables:
- name: name
expression: object.metadata.name
- name: labels
expression: object.metadata.?labels.orValue({})
validations:
- expression: >-
!('purpose' in variables.labels) || variables.name == 'monitoring'
messageExpression: >-
'Namespace ' + variables.name + ': Label purpose ist reserviert (nur monitoring)'
- expression: >-
!variables.name.startsWith('tenant-') || 'tenant' in variables.labels
messageExpression: >-
'Namespace ' + variables.name + ': Mandanten-Namespaces brauchen das Label tenant'
EOF
cat > drift-ns.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: tenant-e
labels: {guard-lab: demo, team: platform, purpose: monitoring}
---
apiVersion: v1
kind: Namespace
metadata:
name: tenant-f
labels: {guard-lab: demo, team: tenant-f, tenent: f}
---
apiVersion: v1
kind: Namespace
metadata:
name: tenant-g
labels: {guard-lab: demo, team: tenant-g, tenant: g}
---
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
labels: {guard-lab: demo, team: platform, purpose: monitoring}
EOF
kyverno apply protect-ns-labels.yaml -r drift-ns.yaml --remove-colorExpected (tested live):
Applying 1 policy rule(s) to 4 resource(s)...
policy protect-ns-labels -> resource /Namespace/tenant-e failed:
1 - Namespace tenant-e: Label purpose ist reserviert (nur monitoring)
policy protect-ns-labels -> resource /Namespace/tenant-f failed:
1 - Namespace tenant-f: Mandanten-Namespaces brauchen das Label tenant
pass: 2, fail: 2, warn: 0, error: 0, skip: 0
tenant-e violates both rules; only the first is reported
(Day 8, Drill 2). The typo tenent in tenant-f
gets caught; monitoring and tenant-g pass. The
policy does not prevent Drill 6: monitoring is not named
tenant-…. For that you need a third rule, such as “never
tenant on namespaces with purpose”.
Save your progress? Sign up for free
Enterprise Note
- These drills belong in a quarterly game day, in staging with real alerts. What gets measured is the time to detection, not just the time to repair.
- For every drill, a runbook with an emergency exit: delete webhooks, create an exception, roll back the policy. The permissions for that sit with a break-glass group, not with everyone.
- Alerts that show these failures early: Hubble drops with
policy-verdict:none,kyverno_admission_requests_total{request_allowed="false"},up == 0per target, missing Kyverno endpoints, growing DNS proxy latency. - Trigger labels (
tenant,purpose) and GeneratingPolicies go through change review like code. Before:kyverno applyagainst real namespaces; after: the target matrix. - Every incident ends with a new test: a matrix line, a
kyverno testcase or a Chainsaw step. Otherwise the same mistake comes back.