Skip to content
Hands-on IT IT Drills

Day 6: Testing and Visualizing Policies

Goal: Treat NetworkPolicies like code: target matrix as a file, check script with a ✔/✘ table, static analysis, Hubble UI and a CI job with kind. · Duration: approx. 90 min · Prerequisite: Day 5

Briefing

  • A policy is only finished when a test proves what it allows and what it forbids. The negative cases are the more important ones.
  • The target matrix belongs in Git as a file (von nach soll: from, to, expected). A script compares it with reality and returns an exit code for CI.
  • Static analysis (policy-assistant) needs no cluster and explains why. Runtime tests (conncheck.sh) prove that the CNI actually does it.
  • Hubble UI and editor.networkpolicy.io make policies and flows visible – good for reviews with teams that do not want to read YAML.
 matrix.tsv ──► np-matrix.sh ──► conncheck.sh ──► kubectl exec ──► ✔/✘ + exit code
 policies/  ──► policy-assistant (offline)    ──► explain and probe tables
 Hubble     ──► UI (service map) / flows.json  ──► editor.networkpolicy.io

Drill 1: Prepare the ground

Continue on cilium-lab. You clear away the Cilium policies from Days 4 and 5, then comes a clean target state for shop – one file per policy.

mkdir -p ~/guard-lab/tag06 ~/guard-lab/code/np-tests/policies
cd ~/guard-lab/code/np-tests
kubectl config use-context kind-cilium-lab
kubectl delete cnp --all -A
kubectl delete ccnp --all
cat > policies/10-default-deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Ingress]
EOF
cat > policies/20-frontend-public.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-public
  namespace: shop
spec:
  podSelector:
    matchLabels: {app: frontend}
  policyTypes: [Ingress]
  ingress:
  - ports:
    - {port: 8080, protocol: TCP}
EOF
cat > policies/30-api-from-frontend.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-frontend
  namespace: shop
spec:
  podSelector:
    matchLabels: {app: api}
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels: {app: frontend}
    ports:
    - {port: 8080, protocol: TCP}
  - from:
    - namespaceSelector:
        matchLabels: {purpose: monitoring}
    ports:
    - {port: 9113, protocol: TCP}
EOF
cat > policies/40-db-from-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-from-api
  namespace: shop
spec:
  podSelector:
    matchLabels: {app: db}
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels: {app: api}
    ports:
    - {port: 5432, protocol: TCP}
EOF
kubectl apply -f policies/

Expected (abridged, may differ depending on version):

ciliumnetworkpolicy.cilium.io "api-l4" deleted
...
ciliumclusterwidenetworkpolicy.cilium.io "tenant-deny" deleted
networkpolicy.networking.k8s.io/default-deny created
networkpolicy.networking.k8s.io/frontend-public created
networkpolicy.networking.k8s.io/api-from-frontend created
networkpolicy.networking.k8s.io/db-from-api created

Why explicit policyTypes, even though the API server would add them? Offline tools (Drill 4) only see the file, not the default added by the server.

Save your progress? Sign up for free

Drill 2: Target matrix and check script

The matrix is a plain text file: source, target, expectation. Think of the negative cases first – what must never work?

cat > matrix.tsv <<'EOF'
# von                 nach                                 soll
default/client        http://frontend.shop                 allow
default/client        http://api.shop                      deny
default/client        tcp://db.shop:5432                   deny
shop/frontend         http://api.shop                      allow
shop/frontend         tcp://db.shop:5432                   deny
shop/api              tcp://db.shop:5432                   allow
monitoring/scraper    http://api.shop:9113/metrics         allow
monitoring/scraper    http://api.shop                      deny
tenant-a/api          http://api.shop                      deny
EOF
cat > np-matrix.sh <<'EOF'
#!/usr/bin/env bash
# np-matrix.sh - test matrix for NetworkPolicies: from -> to, expected vs. actual
#
#   np-matrix.sh [--md] <matrix.tsv>
#
#   matrix.tsv  lines "<ns>/<pod|app> <url> allow|deny", # = comment
#   --md        output as a Markdown table (e.g. for $GITHUB_STEP_SUMMARY)
#
# Exit 0 = everything as expected, 1 = at least one deviation, 2 = usage error.
# CC points to conncheck.sh (default: ~/guard-lab/code/demo/conncheck.sh).
set -uo pipefail

md=false
[[ ${1:-} == --md ]] && { md=true; shift; }
file=${1:-}
cc=${CC:-$HOME/guard-lab/code/demo/conncheck.sh}
if [[ -z $file || ! -r $file ]]; then
  echo "Aufruf: $0 [--md] <matrix.tsv>" >&2
  exit 2
fi

if $md; then
  echo "| von | nach | soll | ist |"
  echo "|---|---|---|---|"
else
  printf '%-22s %-34s %-6s %s\n' VON NACH SOLL IST
fi

ok=0 bad=0
while read -r src url want _; do
  [[ -z ${src:-} || $src == \#* ]] && continue
  "$cc" "$src" "$url" </dev/null >/dev/null 2>&1
  case $? in
    0) got=allow ;;
    1) got=deny ;;
    *) got=error ;;
  esac
  if [[ $got == "$want" ]]; then mark="✔"; ok=$((ok + 1))
  else mark="✘"; bad=$((bad + 1)); fi
  if $md; then
    echo "| \`$src\` | \`$url\` | $want | $mark $got |"
  else
    printf '%-22s %-34s %-6s %s %s\n' "$src" "$url" "$want" "$mark" "$got"
  fi
done < "$file"

echo
echo "$((ok + bad)) Tests: $ok ✔, $bad ✘"
[[ $bad -eq 0 ]]
EOF
chmod +x np-matrix.sh
./np-matrix.sh matrix.tsv; echo "rc=$?"

Expected (abridged, may differ depending on version):

VON                    NACH                               SOLL   IST
default/client         http://frontend.shop               allow  ✔ allow
default/client         http://api.shop                    deny   ✔ deny
default/client         tcp://db.shop:5432                 deny   ✔ deny
shop/frontend          http://api.shop                    allow  ✔ allow
shop/frontend          tcp://db.shop:5432                 deny   ✔ deny
shop/api               tcp://db.shop:5432                 allow  ✔ allow
monitoring/scraper     http://api.shop:9113/metrics       allow  ✔ allow
monitoring/scraper     http://api.shop                    deny   ✔ deny
tenant-a/api           http://api.shop                    deny   ✔ deny

9 Tests: 9 ✔, 0 ✘
rc=0

Two details: </dev/null on the conncheck.sh call prevents a command in the loop from eating the remaining lines of the matrix from standard input. And error (exit 2, e.g. source Pod missing) always counts as a deviation – a test that does not run is red.

Save your progress? Sign up for free

Drill 3: Provoke a regression

A colleague “cleans up labels” and renames frontend to web in the selector.

sed 's/app: frontend}/app: web}/' policies/30-api-from-frontend.yaml \
  | kubectl apply -f -
./np-matrix.sh matrix.tsv > ~/guard-lab/tag06/run.txt; echo "rc=$?"
grep -E '✘|Tests' ~/guard-lab/tag06/run.txt

Expected (abridged, may differ depending on version):

networkpolicy.networking.k8s.io/api-from-frontend configured
rc=1
shop/frontend          http://api.shop                    allow  ✘ deny
9 Tests: 8 ✔, 1 ✘

Exactly one failure, exactly in the right place. The frontend itself keeps running (/ returns 200), only /api/… fails – without a matrix, this is often only noticed in production. Back to the target state:

kubectl apply -f policies/30-api-from-frontend.yaml
./np-matrix.sh matrix.tsv | tail -1

Expected (abridged, may differ depending on version):

networkpolicy.networking.k8s.io/api-from-frontend configured
9 Tests: 9 ✔, 0 ✘

Save your progress? Sign up for free

Drill 4: Static analysis with policy-assistant

policy-assistant from the SIG project network-policy-api is the successor of cyclonus (unmaintained since 2024). It reads policies from files and simulates connections – entirely without a cluster. There is a release binary only for amd64; on Apple Silicon it runs via Rosetta.

cd /tmp
curl -sLO https://github.com/kubernetes-sigs/network-policy-api/releases/download/\
v0.0.1-policy-assistant/policy-assistant_darwin_amd64.tar.gz
tar xzf policy-assistant_darwin_amd64.tar.gz
PA=/tmp/policy-assistant_darwin_amd64/policy-assistant
cd ~/guard-lab/code/np-tests
$PA analyze -v warn --mode explain --policy-path policies/ | cut -c1-86 | head -9

Expected:

explained policies:
+---------+-------------------+-------------------------------+-----------------------
|  TYPE   |      SUBJECT      |         SOURCE RULES          |          PEER
+---------+-------------------+-------------------------------+-----------------------
| Ingress | Namespace:        | [NPv1] shop/api-from-frontend | Namespace:
|         |    shop           |                               |    shop
|         | Pod:              |                               | Pod:
|         |    app = api      |                               |    app = frontend
+         +                   +                               +-----------------------

Warning: policy-assistant only reads the first YAML document per file and requires policyTypes. That is why there is one policy per file (Drill 1).

For the simulation, you describe Pods and namespaces in a JSON file – a small model of the demo app:

cat > probe.json <<'EOF'
{
  "Probes": [
    {"Protocol": "TCP", "Port": 8080},
    {"Protocol": "TCP", "Port": 5432}
  ],
  "Resources": {
    "Namespaces": {
      "shop": {"kubernetes.io/metadata.name": "shop", "guard-lab": "demo"},
      "monitoring": {"kubernetes.io/metadata.name": "monitoring",
                     "purpose": "monitoring"},
      "default": {"kubernetes.io/metadata.name": "default"}
    },
    "Pods": [
      {"Namespace": "shop", "Name": "frontend", "IP": "10.244.1.10",
       "Labels": {"app": "frontend", "tier": "web"},
       "Containers": [{"Name": "nginx", "Port": 8080, "PortName": "http",
                       "Protocol": "TCP"}]},
      {"Namespace": "shop", "Name": "api", "IP": "10.244.1.11",
       "Labels": {"app": "api", "tier": "api"},
       "Containers": [{"Name": "nginx", "Port": 8080, "PortName": "http",
                       "Protocol": "TCP"}]},
      {"Namespace": "shop", "Name": "db", "IP": "10.244.1.12",
       "Labels": {"app": "db", "tier": "data"},
       "Containers": [{"Name": "nginx", "Port": 5432, "PortName": "postgres",
                       "Protocol": "TCP"}]},
      {"Namespace": "default", "Name": "client", "IP": "10.244.2.20",
       "Labels": {"app": "client"},
       "Containers": [{"Name": "curl", "Port": 80, "PortName": "none",
                       "Protocol": "TCP"}]}
    ]
  }
}
EOF
$PA analyze -v warn --mode probe --policy-path policies/ --probe-path probe.json \
  | grep -A8 Combined

Expected:

Combined:
+----------------+----------------+----------+---------+---------------+
|                | DEFAULT/CLIENT | SHOP/API | SHOP/DB | SHOP/FRONTEND |
+----------------+----------------+----------+---------+---------------+
| default/client | N              | X        | N       | .             |
| shop/api       | N              | #        | N       | .             |
| shop/db        | N              | X        | N       | .             |
| shop/frontend  | N              | .        | N       | #             |
+----------------+----------------+----------+---------+---------------+
--
Combined:
...
| default/client | N              | N        | X       | N             |
| shop/api       | N              | N        | .       | N             |
| shop/frontend  | N              | N        | X       | N             |

Rows are sources, columns are targets; one table per probe (8080, then 5432). . = allowed, X = blocked, N = the target does not listen on the port, # = the Pod itself. Same result as in Drill 2 – just in milliseconds and without a cluster. Try the regression from Drill 3 offline: The cell shop/frontend → SHOP/API turns into X.

Tip: policy-assistant knows NetworkPolicy and ANP/BANP v1alpha1, but not yet ClusterNetworkPolicy v1alpha2 and no Cilium types. For those: runtime test.

Save your progress? Sign up for free

Drill 5: Hubble UI – the service map

Back in the cluster. You installed the UI along with everything else on Day 4.

cilium hubble ui --open-browser=false &
sleep 3
for i in 1 2 3; do ~/guard-lab/code/np-tests/np-matrix.sh \
  ~/guard-lab/code/np-tests/matrix.tsv >/dev/null; done

The command sets up a port forward to the Service hubble-ui (default port 12000). Open http://localhost:12000 and select the namespace shop at the top. You see the boxes frontend, api, db and the incoming arrows from client, scraper and tenant-a/api. Dropped flows are marked red; in the flow table below it says dropped with the reason Policy denied. Clicking a flow shows source and destination labels – exactly the labels your selectors have to match.

kill %1

Security: The UI shows all flows of all namespaces. In production, never publish it without authentication (ingress with OIDC) and never via LoadBalancer.

Save your progress? Sign up for free

Drill 6: editor.networkpolicy.io

The policy editor from the Cilium community runs in the browser and understands NetworkPolicy and CiliumNetworkPolicy. It shows a policy as a picture: Pod in the middle, allowed sources and targets on the left and right, default deny as a red area.

cat ~/guard-lab/code/np-tests/policies/30-api-from-frontend.yaml | pbcopy
  1. Open https://editor.networkpolicy.io, choose “Upload” or the YAML field, and paste.
  2. Check: Ingress shows app=frontend on 8080 and purpose=monitoring on 9113.
  3. Egress is open (green), because the policy only governs Ingress.
  4. Make changes in the picture and download the YAML again.

The editor can also generate policy suggestions from Hubble flows:

hubble observe -P --namespace shop --last 200 -o json > ~/guard-lab/tag06/flows.json
wc -l ~/guard-lab/tag06/flows.json

Expected (abridged, may differ depending on version):

200 /Users/<du>/guard-lab/tag06/flows.json

Security: Flows contain IPs, names and URLs. You do not upload real customer flows to a public web tool. For production: generate suggestions locally and check them in a review – a generated allow-all-observed is not a security architecture.

Save your progress? Sign up for free

Drill 7: CI job with kind

What runs locally also runs in the pipeline: a fresh kind cluster (kindnet enforces NetworkPolicies), demo app, policies, matrix. Assumption: Your repo contains code/demo/ and code/np-tests/ like ~/guard-lab/code/.

mkdir -p ~/guard-lab/code/np-tests/ci && cd ~/guard-lab/code/np-tests/ci
cat > github-np-tests.yaml <<'EOF'
# Day 6: GitHub Actions - copy to .github/workflows/np-tests.yaml
name: np-tests
on:
  pull_request:
    paths: ["code/**"]
  workflow_dispatch:
jobs:
  matrix:
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    defaults:
      run:
        shell: bash               # = bash -eo pipefail: tee does not swallow the rc
    steps:
    - uses: actions/checkout@v7
    - uses: helm/[email protected]      # kindnet enforces NetworkPolicies
      with:
        version: v0.33.0
        cluster_name: np-ci
    - name: Demo-App ausrollen
      run: |
        kubectl apply -f code/demo/
        kubectl wait --for=condition=Ready pod --all -A --timeout=300s
    - name: Policies anwenden
      run: kubectl apply -f code/np-tests/policies/
    - name: Testmatrix
      env:
        CC: ${{ github.workspace }}/code/demo/conncheck.sh
      run: |
        sleep 5
        code/np-tests/np-matrix.sh --md code/np-tests/matrix.tsv \
          | tee -a "$GITHUB_STEP_SUMMARY"
EOF

Copied to .github/workflows/np-tests.yaml, the job runs on every pull request that changes code/. --md writes the table into the job summary – reviewers see ✔/✘ directly in the PR. shell: bash is needed so that pipefail applies and tee does not swallow the exit code of np-matrix.sh.

Check the workflow locally with actionlint before you push:

go run github.com/rhysd/actionlint/cmd/actionlint@latest github-np-tests.yaml \
  && echo "✔ actionlint"

Expected:

✔ actionlint

In GitLab, kind runs in Docker-in-Docker. The API server then does not sit on the job’s 127.0.0.1, but in the service container docker – hence the separate kind configuration:

cat > kind-ci.yaml <<'EOF'
# Day 6: kind in GitLab CI (Docker-in-Docker) - API server via the service "docker"
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: "0.0.0.0"   # only acceptable in a throwaway CI job
  apiServerPort: 6443
kubeadmConfigPatches:
- |
  kind: ClusterConfiguration
  apiServer:
    certSANs: ["docker"]
EOF
cat > gitlab-np-tests.yaml <<'EOF'
# Day 6: GitLab CI - add as a job to .gitlab-ci.yml (runner with privileged dind)
np-tests:
  image: docker:29.1
  services:
  - name: docker:29.1-dind
    alias: docker
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    KUBECTL_URL: https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl
  rules:
  - changes: ["code/**/*"]
  before_script:
  - apk add --no-cache bash curl
  - curl -sLo /usr/local/bin/kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-amd64
  - curl -sLo /usr/local/bin/kubectl "$KUBECTL_URL"
  - chmod +x /usr/local/bin/kind /usr/local/bin/kubectl
  script:
  - kind create cluster --name np-ci --config code/np-tests/ci/kind-ci.yaml
  - sed -i 's#https://0.0.0.0:6443#https://docker:6443#' ~/.kube/config
  - kubectl apply -f code/demo/
  - kubectl wait --for=condition=Ready pod --all -A --timeout=300s
  - kubectl apply -f code/np-tests/policies/
  - sleep 5
  - CC=$PWD/code/demo/conncheck.sh code/np-tests/np-matrix.sh code/np-tests/matrix.tsv
EOF

You add the job to your .gitlab-ci.yml. The runner must run privileged (for docker:dind) – that belongs on a dedicated runner, not on the shared one. The GitLab job has not been tested live; the GitHub workflow has been checked with actionlint.

Tip: If you want to test Cilium policies in CI, install Cilium into the kind cluster via Helm as on Day 4. Plan for 3–5 minutes of additional runtime.

Save your progress? Sign up for free

Checkpoint

cd ~/guard-lab/code/np-tests
PA=${PA:-/tmp/policy-assistant_darwin_amd64/policy-assistant}
[ "$(ls policies/*.yaml | wc -l)" -ge 4 ] \
  && echo "✔ Policies einzeln abgelegt" || echo "✘ policies/ unvollständig"
grep -q deny matrix.tsv && echo "✔ Matrix mit Negativ-Fällen" \
  || echo "✘ keine deny-Zeilen in der Matrix"
./np-matrix.sh matrix.tsv >/dev/null && echo "✔ Matrix grün" || echo "✘ Matrix rot"
./np-matrix.sh >/dev/null 2>&1; [ $? -eq 2 ] \
  && echo "✔ Aufruffehler = Exit 2" || echo "✘ Exit-Code bei Aufruffehler"
$PA analyze -v warn --mode probe --policy-path policies/ --probe-path probe.json \
  | grep -q 'shop/frontend  | N              | .' \
  && echo "✔ policy-assistant: frontend -> api erlaubt" || echo "✘ policy-assistant"

Challenge

Add an egress case to the matrix: The database must not reach the internet (shop/db https://example.com deny). Let the matrix turn red and make it green again with a new policy file, without breaking any other lines.

Hint: An egress policy for db without any allowed targets is enough – db itself does not open any connections.

Solution

cd ~/guard-lab/code/np-tests
echo "shop/db               https://example.com                  deny" >> matrix.tsv
./np-matrix.sh matrix.tsv | tail -1
cat > policies/50-db-no-egress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-no-egress
  namespace: shop
spec:
  podSelector:
    matchLabels: {app: db}
  policyTypes: [Egress]
EOF
kubectl apply -f policies/50-db-no-egress.yaml
./np-matrix.sh matrix.tsv | tail -1

Expected (abridged, may differ depending on version):

10 Tests: 9 ✔, 1 ✘
networkpolicy.networking.k8s.io/db-no-egress created
10 Tests: 10 ✔, 0 ✘

The db’s responses to the api keep flowing: NetworkPolicies are stateful; return packets of an allowed connection do not need an egress rule.

Save your progress? Sign up for free

Enterprise Note

  • The matrix is a contract with the teams: Changes to it go through review like code. One file per tenant, shared platform cases in a separate one.
  • Runtime tests belong in the pipeline after every CNI upgrade and every platform rollout (smoke test namespace with a client Pod in the real cluster).
  • Static analysis before the merge, runtime test after the deploy – both, not one.
  • Export Hubble flows to the SIEM and alert on policy-verdict:none … DENIED: Those are the connections for which someone forgot a policy.
  • Clean up: kind delete cluster --name cilium-lab; from Day 7 on, you continue with kyverno-lab.