Skip to content
Hands-on IT IT Drills

Day 0: Training Ground

Goal: Check your tools, start the kind cluster np-lab, roll out the demo application and test the first connection with conncheck.sh. · Duration: approx. 45 min · Prerequisite: none

Briefing

  • Three kind clusters, never all at the same time: np-lab (Part A), cilium-lab (Part B), kyverno-lab (Part C/D).
  • One demo application for everything: shop with frontend → api → db, plus tenant-a, tenant-b, monitoring and a neutral client.
  • One image for all servers: nginxinc/nginx-unprivileged:1.31-alpine (small, with curl, wget, nc). Test Pods use curlimages/curl:8.22.0.
  • conncheck.sh tests every connection from inside a Pod and answers with ✔ allowed or ✘ blocked.
~/guard-lab/
├── code/demo/      demo app (00-…40-*.yaml) + conncheck.sh
├── kind/           np-lab.yaml, cilium-lab.yaml, kyverno-lab.yaml
└── tag01/ …        your working folders per day

Drill 1: Check your tools

docker info --format '{{.ServerVersion}} {{.MemTotal}}'
kind version
kubectl version --client | head -1
helm version --short

Expected (abridged, may differ depending on version):

29.5.2 8307167232
kind v0.33.0 go1.27.0 darwin/arm64
Client Version: v1.37.1
v4.3.0+gbec5b06

You only need the Kyverno CLI and cosign from Day 7 and Day 10 on:

kyverno version | head -1
cosign version | grep GitVersion

Expected (abridged):

Version: 1.19.1
GitVersion:    v3.1.3

Tip: Give Docker at least 6 GB of RAM. Every kind node is a container. A node with the demo app needs about 1 GB, with Kyverno or Cilium more like 2 GB.

Save your progress? Sign up for free

Drill 2: Folders and demo code

mkdir -p ~/guard-lab/{code,kind,tag01,tag02,tag03}
git clone https://github.com/thomaszachmann/kyverno-networkpolicies-zum-anfassen.git \
  ~/kyverno-networkpolicies-zum-anfassen 2>/dev/null \
  || git -C ~/kyverno-networkpolicies-zum-anfassen pull
cp -R ~/kyverno-networkpolicies-zum-anfassen/demo ~/guard-lab/code/
chmod +x ~/guard-lab/code/demo/conncheck.sh
ls ~/guard-lab/code/demo

Expected:

00-namespaces.yaml  10-shop.yaml  20-tenants.yaml  30-monitoring.yaml
40-client.yaml      conncheck.sh

The numbers make sure that kubectl apply -f <ordner> creates the namespaces first. A look at the labels you will need constantly from Day 1 on:

cd ~/guard-lab/code/demo
grep -h -A4 'labels:' 00-namespaces.yaml | grep -v labels
grep -h 'labels: {app' 10-shop.yaml | sort -u

Expected (abridged):

    guard-lab: demo
    team: shop
    tenant: a
    purpose: monitoring
  labels: {app: api, tier: api}
  labels: {app: db, tier: data}
  labels: {app: frontend, tier: web}
Namespace Workload Labels Ports
shop frontend app=frontend, tier=web 8080 http
shop api app=api, tier=api 8080 http, 9113 metrics
shop db app=db, tier=data 5432 postgres
tenant-a/-b api app=api, tier=api 8080 http, 9113 metrics
monitoring scraper app=scraper, tier=monitoring –
default client app=client –

The Services are named like the Deployments. As Services, api and frontend listen on port 80, db on 5432. db is a simulated database: nginx on port 5432. That saves a 100 MB image, and for NetworkPolicies only the port matters.

Save your progress? Sign up for free

Drill 3: Three kind configurations

cd ~/guard-lab/kind
cat > np-lab.yaml <<'YAML'
# Part A: kindnet enforces NetworkPolicies itself (since kind 0.24)
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: np-lab
nodes:
- role: control-plane
- role: worker
- role: worker
YAML
cat > cilium-lab.yaml <<'YAML'
# Part B: no default CNI, no kube-proxy - Cilium takes over both (Day 4)
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: cilium-lab
networking:
  disableDefaultCNI: true
  kubeProxyMode: none
nodes:
- role: control-plane
- role: worker
- role: worker
YAML
cat > kyverno-lab.yaml <<'YAML'
# Part C/D: Kyverno + local registry kind-registry:5000 (Day 10)
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: kyverno-lab
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"
nodes:
- role: control-plane
- role: worker
- role: worker
YAML
ls

Expected:

cilium-lab.yaml  kyverno-lab.yaml  np-lab.yaml

Why the containerdConfigPatches block? It makes containerd read registry mirrors from /etc/containerd/certs.d. On Day 10 you only drop a hosts.toml there instead of rebuilding the cluster.

Tip: Low on RAM? Delete the two worker lines. All drills in Part A also run on a single node. This chapter was tested single-node.

Save your progress? Sign up for free

Drill 4: Start np-lab

kind create cluster --config ~/guard-lab/kind/np-lab.yaml
kubectl get nodes
kubectl -n kube-system get ds kindnet \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected (abridged, may differ depending on version):

 ✓ Ensuring node image (kindest/node:v1.37.0)
 ✓ Installing CNI
Set kubectl context to "kind-np-lab"
NAME                   STATUS   ROLES           AGE   VERSION
np-lab-control-plane   Ready    control-plane   40s   v1.37.0
np-lab-worker          Ready    <none>          20s   v1.37.0
np-lab-worker2         Ready    <none>          20s   v1.37.0
docker.io/kindest/kindnetd:v20260820-69b56db7

kindnet is kind’s default CNI. Since kind 0.24 it ships its own NetworkPolicy controller (kube-network-policies). Drill 7 proves whether it really takes effect.

Save your progress? Sign up for free

Drill 5: Roll out the demo application

kubectl apply -f ~/guard-lab/code/demo/
kubectl wait --for=condition=Ready pod --all -A --timeout=300s
kubectl get pods -A -l 'app in (frontend,api,db,scraper,client)'

Expected (abridged):

namespace/shop created
namespace/tenant-a created
...
deployment.apps/scraper created
pod/client created
NAMESPACE    NAME                        READY   STATUS    RESTARTS   AGE
default      client                      1/1     Running   0          60s
monitoring   scraper-7c99db6d96-rw2gv    1/1     Running   0          60s
shop         api-5ccd4587ff-gwhbn        1/1     Running   0          60s
shop         db-5d6755ff6b-z5vv9         1/1     Running   0          60s
shop         frontend-5f9bb6b549-8hpqx   1/1     Running   0          60s
tenant-a     api-5ccd4587ff-2x5n2        1/1     Running   0          60s
tenant-b     api-5ccd4587ff-f48wx        1/1     Running   0          60s

The first start takes a few minutes because every node pulls the images. Now the chain from the outside in:

kubectl exec client -- curl -s http://frontend.shop/
kubectl exec client -- curl -s http://frontend.shop/api/orders
kubectl exec client -- curl -s http://api.tenant-a/x
kubectl -n monitoring logs deploy/scraper --tail=3

Expected:

frontend ok
{"svc":"api","ns":"shop","path":"/api/orders"}
{"svc":"api","ns":"tenant-a","path":"/x"}
ok   shop
ok   tenant-a
ok   tenant-b

The frontend passes /api/… on to api.shop and resolves the name via DNS on every request. That becomes important on Day 2. Every 10 s the scraper fetches /metrics (port 9113) from all three api Pods and writes ok or FAIL to the log.

Save your progress? Sign up for free

Drill 6: Get to know conncheck.sh

The script does exactly one thing: open a connection from inside a source Pod and report the result unambiguously.

cd ~/guard-lab/code/demo
sed -n '2,11p' conncheck.sh

Expected:

# conncheck.sh - testet eine Verbindung aus einem Pod heraus.
#
#   conncheck.sh <ns>/<pod|app> <url> [allow|deny]
#
#   <ns>/<pod|app>  Quelle: Pod-Name oder Wert des Labels app (erster Running-Pod)
#   <url>           http://host[:port][/pfad]  oder  tcp://host:port
#   allow|deny      optional: Erwartung. Dann Exit 0 = wie erwartet, 1 = Abweichung
#
# Ohne Erwartung: Exit 0 = erlaubt, 1 = blockiert, 2 = Aufruffehler.

The core: kubectl exec into the source Pod, and there curl (HTTP) or nc -z (TCP) with a 2 s timeout. The curl exit code is translated: 6 = DNS error, 7 = refused, 28 = timeout. An HTTP status like 404 or 502 counts as allowed, because the connection was established.

alias cc=~/guard-lab/code/demo/conncheck.sh
cc shop/frontend http://api.shop
cc default/client tcp://db.shop:5432
cc default/client http://nix.shop.svc
cc shop/frontend http://api.shop:81
cc shop/nope http://x; echo "rc=$?"

Expected:

✔ erlaubt   shop/frontend -> http://api.shop (HTTP 200)
✔ erlaubt   default/client -> tcp://db.shop:5432
✘ blockiert default/client -> http://nix.shop.svc (DNS-Fehler)
✘ blockiert shop/frontend -> http://api.shop:81 (Timeout)
✘ Quelle shop/nope nicht gefunden
rc=2

Watch out for line 4: The Service api has no port 81, so the packet goes nowhere. conncheck.sh reports “blockiert” (blocked) even though no policy is involved at all. A ✘ only means: no connection. Whether a policy is to blame is something you check with a cross-check.

With an expectation, the script becomes a test:

cc default/client http://api.shop deny; echo "rc=$?"

Expected:

✔ erlaubt   default/client -> http://api.shop (HTTP 200)  <-- erwartet: deny
rc=1

Tip: TIMEOUT=5 cc … extends the wait time in case your laptop is under load.

Save your progress? Sign up for free

Drill 7: Proof – kindnet enforces policies

A NetworkPolicy without an enforcing CNI is just YAML. So prove it first:

cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: probe-deny
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
  - Ingress
EOF
cc default/client http://frontend.shop
kubectl -n shop delete networkpolicy probe-deny
cc default/client http://frontend.shop

Expected:

networkpolicy.networking.k8s.io/probe-deny created
✘ blockiert default/client -> http://frontend.shop (Timeout)
networkpolicy.networking.k8s.io "probe-deny" deleted from shop namespace
✔ erlaubt   default/client -> http://frontend.shop (HTTP 200)

No ✘ on the first call? Then your CNI does not enforce policies. Old kind version or a different CNI: Install Calico or go straight to Cilium (Day 4).

Save your progress? Sign up for free

Checkpoint

cc=~/guard-lab/code/demo/conncheck.sh
k() { kubectl get "$@" >/dev/null 2>&1; }
kubectl config current-context | grep -q kind-np-lab \
  && echo "✔ Kontext kind-np-lab" || echo "✘ falscher Kontext"
for ns in shop tenant-a tenant-b monitoring; do
  k ns "$ns" && echo "✔ Namespace $ns" || echo "✘ Namespace $ns fehlt"
done
[ "$(kubectl get netpol -A --no-headers 2>/dev/null | wc -l)" -eq 0 ] \
  && echo "✔ keine NetworkPolicies" || echo "✘ alte NetworkPolicies vorhanden"
$cc default/client http://frontend.shop allow >/dev/null \
  && echo "✔ client -> frontend" || echo "✘ client -> frontend"
$cc shop/api tcp://db.shop:5432 allow >/dev/null \
  && echo "✔ api -> db" || echo "✘ api -> db"
ls ~/guard-lab/kind/*.yaml | wc -l | grep -q 3 \
  && echo "✔ drei kind-Configs" || echo "✘ kind-Configs fehlen"

Challenge

Write ~/guard-lab/code/matrix.sh. From the sources default/client, shop/frontend and tenant-a/api it checks the targets frontend.shop, api.shop, tcp://db.shop:5432 and api.tenant-b and prints all results. You will need it from Day 1 on after every new policy.

Hint: Two nested for loops and conncheck.sh without an expectation are enough.

Solution

cat > ~/guard-lab/code/matrix.sh <<'EOF'
#!/usr/bin/env bash
# matrix.sh - all sources against all targets (lab test matrix)
cc=~/guard-lab/code/demo/conncheck.sh
srcs="default/client shop/frontend tenant-a/api"
dsts="http://frontend.shop http://api.shop tcp://db.shop:5432 http://api.tenant-b"
for s in $srcs; do
  for d in $dsts; do
    "$cc" "$s" "$d"
  done
done
EOF
chmod +x ~/guard-lab/code/matrix.sh
~/guard-lab/code/matrix.sh | head -4

Expected (abridged):

✔ erlaubt   default/client -> http://frontend.shop (HTTP 200)
✔ erlaubt   default/client -> http://api.shop (HTTP 200)
✔ erlaubt   default/client -> tcp://db.shop:5432
✔ erlaubt   default/client -> http://api.tenant-b (HTTP 200)

Without policies everything is allowed: the Kubernetes network is flat and open by default. That is exactly what you will change starting tomorrow.

Save your progress? Sign up for free

Enterprise Note

  • At a customer site, first check which CNI is running and whether it enforces NetworkPolicies (Flannel alone does not; Canal, Calico, Cilium, OVN-Kubernetes do).
  • RKE2 ships Canal, OpenShift ships OVN-Kubernetes. Both enforce policies, but the details (logging, host traffic) differ.
  • In production, images come from an internal registry with a digest instead of a tag (Day 10).
  • A check script like conncheck.sh belongs in the pipeline after every rollout (Day 6).
  • Delete the lab cluster after the day: kind delete cluster --name np-lab.