Skip to content
Hands-on IT IT Drills

Day 11: Reachable from Outside, Sealed Off Inside

Goal: Make notes reachable through a Gateway at http://notes.localhost:30080, limit the traffic in the namespace to what is necessary with NetworkPolicies, and look inside the api’s scratch container. · Duration: approx. 120 min · Prerequisite: Day 10

Briefing

  • Gateway API separates roles: the platform provides a GatewayClass (which controller, which parameters), the app team creates the Gateway (listener: port, hostname) and the HTTPRoute (host, path → Service).
  • The controller here is Envoy Gateway: for each Gateway it starts an Envoy Pod plus a Service in the namespace envoy-gateway-system.
  • A NetworkPolicy selects Pods by label. As soon as a policy selects a Pod, only what a policy allows is permitted for it.
  • kubectl debug attaches an ephemeral container with tools to a running Pod – even to one without a shell.
 Mac: curl http://notes.localhost:30080
        │  extraPortMappings (Day 0)
        ▼
 Node :30080 ─► Service envoy-… (NodePort) ─► Pod envoy    envoy-gateway-system
                                                │ Gateway notes, HTTPRoute web
                                                ▼ Host: notes.localhost
                          Service web (ClusterIP) ─► web ─► api ─► db    notes

Drill 1: Start of the Day and Envoy Gateway

source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag11 && cd ~/k8s-lab/tag11
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag10/notes/." .
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.3 \
  -n envoy-gateway-system --create-namespace --wait | grep -E '^(NAME|STATUS):'
kubectl -n envoy-gateway-system get pods
kubectl get crd | grep -c gateway.networking.k8s.io
kubectl get crd gateways.gateway.networking.k8s.io \
  -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}'
echo

Expected (abridged, names, IPs and times will differ on your machine):

Pulled: docker.io/envoyproxy/gateway-helm:v1.9.3
Digest: sha256:be1b8f16…
NAME: eg
STATUS: deployed
NAME                             READY   STATUS      RESTARTS   AGE
eg-gateway-helm-certgen-xbb8q    0/1     Completed   0          42s
envoy-gateway-6776489b4b-rnzg5   1/1     Running     0          19s
10
v1.6.1

The chart ships the Gateway API CRDs (here bundle v1.6.1) and Envoy Gateway’s own CRDs. The certgen Job generates the controller’s certificates once. This takes just under a minute.

Save your progress? Sign up for free

Drill 2: GatewayClass – the Platform Side

The GatewayClass eg refers to an EnvoyProxy object with parameters: the Envoy Service should be a NodePort on 30080, because that is the only port kind forwards to the Mac. First draft:

cat > k8s/gateway.yaml <<'EOF'
# k8s/gateway.yaml – first draft (Day 11, Drill 2; finished in Drill 4)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: nodeport
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
        patch:
          type: StrategicMerge
          value:
            spec:
              ports:
                - port: 80
                  nodePort: 30080
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: nodeport
    namespace: envoy-gateway-system
EOF
kubectl apply --dry-run=server -f k8s/gateway.yaml
kubectl apply -f k8s/gateway.yaml
kubectl get gatewayclass

Expected:

envoyproxy.gateway.envoyproxy.io/nodeport created (server dry run)
gatewayclass.gateway.networking.k8s.io/eg created (server dry run)
envoyproxy.gateway.envoyproxy.io/nodeport created
gatewayclass.gateway.networking.k8s.io/eg created
NAME   CONTROLLER                                      ACCEPTED   AGE
eg     gateway.envoyproxy.io/gatewayclass-controller   True       0s

ACCEPTED True means: the controller considers itself responsible. There is no Envoy Pod yet – it is only created with the first Gateway. Like namespace.yaml, gateway.yaml belongs to the platform part in k8s/, not in the chart.

Save your progress? Sign up for free

Drill 3: Chart 0.2.0 – Gateway and Route

The app team delivers the Gateway and HTTPRoute in the chart. Fetch the state from the companion code and look at the changes:

cp -R "$LAB/code/tag11/notes/chart/." chart/
grep -E '^version' chart/notes/Chart.yaml
sed -n '/^gateway:/,$p' chart/notes/values.yaml
grep -A1 '^  service:' chart/notes/values.yaml

Expected:

version: 0.2.0
gateway:
  enabled: true
  className: eg
  hostname: notes.localhost

networkPolicy:
  enabled: true
  gatewayNamespace: envoy-gateway-system
  service:
    type: ClusterIP   # ab Tag 11 kommt der Verkehr über das Gateway

The new template (the NetworkPolicies from the same delivery follow in Drill 6):

# chart/notes/templates/gateway.yaml – state at end of Day 11
{{- if .Values.gateway.enabled }}
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: notes
  labels:
    {{- include "notes.labels" (dict "ctx" $ "name" "gateway") | nindent 4 }}
spec:
  gatewayClassName: {{ .Values.gateway.className }}
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      hostname: {{ .Values.gateway.hostname | quote }}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web
  labels:
    {{- include "notes.labels" (dict "ctx" $ "name" "web") | nindent 4 }}
spec:
  parentRefs:
    - name: notes
  hostnames:
    - {{ .Values.gateway.hostname | quote }}
  rules:
    - backendRefs:
        - name: web
          port: 3000
{{- end }}
  • The listener only accepts requests for notes.localhost on port 80. Port 80 is the port of the Envoy Service; the NodePort 30080 comes from the EnvoyProxy patch.
  • The route attaches itself to the Gateway via parentRefs and sends everything to the Service web. Without matches it applies to every path.
  • web becomes ClusterIP: from now on, the NodePort 30080 belongs to the Gateway.
helm lint chart/notes | tail -1
helm template notes chart/notes -n notes \
  | kubectl apply --dry-run=server -f - 2>/dev/null | grep created
helm upgrade notes chart/notes -n notes --wait | tail -3
kubectl get svc/web gateway/notes httproute/web

Expected (abridged):

1 chart(s) linted, 0 chart(s) failed
networkpolicy.networking.k8s.io/default-deny-ingress created (server dry run)
networkpolicy.networking.k8s.io/web-from-gateway created (server dry run)
networkpolicy.networking.k8s.io/api-from-clients created (server dry run)
networkpolicy.networking.k8s.io/db-from-api created (server dry run)
gateway.gateway.networking.k8s.io/notes created (server dry run)
httproute.gateway.networking.k8s.io/web created (server dry run)
pod/notes-test-api created (server dry run)
notes 0.2.0 ist im Namespace notes installiert.
Über das Gateway: http://notes.localhost:30080
Test: helm test notes -n notes
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/web   ClusterIP   10.96.196.80   <none>        3000/TCP   4m

NAME                                      CLASS   ADDRESS      PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/notes   eg      172.18.0.6   True         26s

NAME                                      HOSTNAMES             AGE
httproute.gateway.networking.k8s.io/web   ["notes.localhost"]   26s

2>/dev/null hides harmless messages (missing last-applied annotation on the Helm objects, field is immutable for the hook Job as on Day 10). Because the chart version in the labels changes, all Pods roll once; api may restart once or twice along the way until db is ready again (Day 7).

Save your progress? Sign up for free

Drill 4: The externalTrafficPolicy Trap

PROGRAMMED True – so let’s go:

curl -sS -m 5 http://notes.localhost:30080/healthz; echo "Exit: $?"

Expected:

curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
Exit: 28

The connection is established, but nobody answers. Where does Envoy run, and how is its Service configured?

kubectl -n envoy-gateway-system get pods \
  -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName'
c='NAME:.metadata.name,TYPE:.spec.type'
kubectl -n envoy-gateway-system get svc \
  -l gateway.envoyproxy.io/owning-gateway-name=notes \
  -o custom-columns="$c,POLICY:.spec.externalTrafficPolicy"

Expected (abridged):

NAME                                          NODE
envoy-gateway-6776489b4b-rnzg5                lab-worker
envoy-notes-notes-3aa9e90e-646c5fd4cc-88ss8   lab-worker2
NAME                         TYPE       POLICY
envoy-notes-notes-3aa9e90e   NodePort   Local

Envoy Gateway sets externalTrafficPolicy: Local: a node only forwards traffic from outside to Pods on itself. With kind, however, port 30080 always arrives at the control plane node, and no Envoy runs there. With Cluster, every node may forward to any Envoy Pod. Add two lines to the EnvoyProxy patch:

# k8s/gateway.yaml (excerpt) – state at end of Day 11
            spec:
              # The default is Local: then only a node on which Envoy runs answers.
              externalTrafficPolicy: Cluster
              ports:
                - port: 80
                  nodePort: 30080

The complete file is in code/tag11/notes/k8s/gateway.yaml:

cp "$LAB/code/tag11/notes/k8s/gateway.yaml" k8s/
kubectl diff -f k8s/gateway.yaml | grep -E '^[-+] '
kubectl apply -f k8s/gateway.yaml
kubectl -n envoy-gateway-system get svc \
  -l gateway.envoyproxy.io/owning-gateway-name=notes \
  -o custom-columns='NAME:.metadata.name,POLICY:.spec.externalTrafficPolicy'
curl -sS -m 5 http://notes.localhost:30080/healthz; echo
curl -s http://notes.localhost:30080/api/notes | jq -c 'map(.text)'

Expected (abridged, the notes come from Day 10):

-  generation: 1
+  generation: 2
+              externalTrafficPolicy: Cluster
envoyproxy.gateway.envoyproxy.io/nodeport configured
gatewayclass.gateway.networking.k8s.io/eg unchanged
NAME                         POLICY
envoy-notes-notes-3aa9e90e   Cluster
ok
["Postgres eins","Postgres zwei","Postgres drei","Kontrollpunkt 8"]

Envoy Gateway applies the change to the EnvoyProxy to the Service immediately. The notes now come via Envoy → web → api → Postgres. http://notes.localhost:30080 also works in the browser.

Tip: curl on the Mac resolves *.localhost to 127.0.0.1 by itself, without /etc/hosts. If it doesn’t on your machine (older curl versions, other tools), set the Host header by hand: curl -H 'Host: notes.localhost' localhost:30080/… or curl --resolve notes.localhost:30080:127.0.0.1 http://notes.localhost:30080/….

A test on the node itself (docker exec lab-control-plane curl … 127.0.0.1:30080) would have returned ok even before – kube-proxy treats traffic from the node itself differently. That is why a test on the node is deceptive: always check the path that the user takes too.

Save your progress? Sign up for free

Drill 5: Hostnames and 404

The Gateway decides based on the Host header:

curl -s -o /dev/null -w '%{http_code}\n' localhost:30080/healthz
curl -si -H 'Host: falsch.localhost' localhost:30080/healthz
curl -si -H 'Host: notes.localhost' localhost:30080/healthz | head -2
curl -s http://notes.localhost:30080/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"über das Gateway"}'; echo

Expected (abridged):

404
HTTP/1.1 404 Not Found
date: Fri, 30 Oct 2026 15:09:03 GMT
content-length: 0

HTTP/1.1 200 OK
x-powered-by: Express
{"id":"68be9f2b-…","text":"über das Gateway"}

For localhost and falsch.localhost there is neither a listener nor a route, so Envoy itself answers with an empty 404. The 200 with x-powered-by: Express comes from notes-web. The state of the Gateway and the route is shown by their conditions:

kubectl get gateway notes \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}'; echo
kubectl get httproute web \
  -o jsonpath='{range .status.parents[0].conditions[*]}{.type}={.status} {end}'; echo

Expected:

Accepted=True Programmed=True
Accepted=True ResolvedRefs=True

ResolvedRefs=False would mean: the Service in backendRefs doesn’t exist or the port is wrong. That is the first place you look when you get a 404 or 500.

Save your progress? Sign up for free

Drill 6: NetworkPolicy – Close Everything, Then Open Selectively

The upgrade from Drill 3 also created four NetworkPolicies. kindnet, kind’s Pod network, enforces them. The template (excerpt, complete in code/tag11/notes/chart/notes/templates/networkpolicy.yaml):

# chart/notes/templates/networkpolicy.yaml (excerpt) – state at end of Day 11
# 1. Deny all incoming traffic in the namespace …
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes: [Ingress]
---
# 2. … web only from the Gateway
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-from-gateway
spec:
  podSelector:
    matchLabels: {app.kubernetes.io/name: web}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: {{ .Values.networkPolicy.gatewayNamespace }}
        - podSelector:
            matchLabels: {app.kubernetes.io/name: test}
      ports:
        - port: 3000
  • podSelector: {} selects all Pods in the namespace; without an ingress rule, all incoming traffic is thereby denied.
  • Policies add up: web may additionally be addressed from envoy-gateway-system and by the Helm test Pod. Two entries under from are an or.
  • Following the same pattern: api-from-clients (api only from web, worker, test, port
    1. and db-from-api (db only from api and migrate, port 5432).
  • Outgoing traffic (Egress) stays open; so DNS keeps working.
kubectl get networkpolicy
kubectl describe networkpolicy api-from-clients | sed -n '/^Spec/,$p'

Expected (abridged):

NAME                   POD-SELECTOR                 AGE
api-from-clients       app.kubernetes.io/name=api   2m38s
db-from-api            app.kubernetes.io/name=db    2m38s
default-deny-ingress   <none>                       2m38s
web-from-gateway       app.kubernetes.io/name=web   2m38s
Spec:
  PodSelector:     app.kubernetes.io/name=api
  Allowing ingress traffic:
    To Port: 8080/TCP
    From:
      PodSelector: app.kubernetes.io/name in (test,web,worker)
  Not affecting egress traffic
  Policy Types: Ingress

Save your progress? Sign up for free

Drill 7: Proof with a Foreign Pod

A policy you haven’t tested is a guess. A Pod in the namespace default plays the intruder:

kubectl -n default run fremd --image=busybox:1.37 --restart=Never -- sleep 3600
kubectl -n default wait --for=condition=Ready pod/fremd --timeout=120s
kubectl -n default exec fremd -- wget -qO- -T 3 http://api.notes:8080/version
kubectl -n default exec fremd -- wget -qO- -T 3 http://web.notes:3000/healthz

Expected:

pod/fremd created
pod/fremd condition met
wget: download timed out
command terminated with exit code 1
wget: download timed out
command terminated with exit code 1

No Connection refused as with an empty Service (Day 3), but a timeout: the packets are silently dropped. Now the allowed and forbidden paths from the inside:

kubectl exec deploy/web -- wget -qO- -T 3 http://api:8080/version
kubectl exec deploy/web -- nc -z -w 3 db 5432; echo "web → db: Exit $?"
kubectl logs deploy/worker --tail 1
kubectl get pods
helm test notes -n notes | grep Phase

Expected (abridged):

0.2.0
command terminated with exit code 1
web → db: Exit 1
summary: 5 notes, 11 words
NAME                      READY   STATUS      RESTARTS       AGE
api-646785c9b8-46xbl      1/1     Running     0              95s
api-646785c9b8-fs2l9      1/1     Running     2 (108s ago)   113s
api-646785c9b8-hrrmd      1/1     Running     0              91s
db-0                      1/1     Running     0              110s
…
web-7955fff6b-wjv6q       1/1     Running     0              113s
worker-847b98cbbd-dn8fp   1/1     Running     0              112s
Phase:          Succeeded

web may talk to the api, but not to the database; worker keeps counting; all Pods stay Ready. The kubelet’s probes are exempt from NetworkPolicies. You still need the Pod fremd in the challenge.

Save your progress? Sign up for free

Drill 8: kubectl debug Under restricted

notes-api is scratch: no shell, no ps (Day 1).

POD=$(kubectl get pods -l app.kubernetes.io/name=api -o name | head -1)
kubectl exec "$POD" -- sh

Expected (abridged):

error: Internal error occurred: … exec: "sh": executable file not found in $PATH

So a debug container with busybox that sees the processes of the container api:

kubectl debug -it "$POD" --image=busybox:1.37 --target=api -- sh

Expected (abridged):

Targeting container "api". If you don't see processes from this container it may …
Defaulting debug container name to debugger-js86b.
Error from server (Forbidden): pods "api-646785c9b8-46xbl" is forbidden: violates
PodSecurity "restricted:latest": allowPrivilegeEscalation != false (…), unrestricted
capabilities (… must not include "SYS_PTRACE" in securityContext.capabilities.add)

The default profile general adds SYS_PTRACE – which restricted forbids (Day 8). The profile restricted sets the appropriate fields:

kubectl debug -it "$POD" --image=busybox:1.37 --target=api --profile=restricted -- sh

Expected (abridged; kubectl waits afterwards, abort with Ctrl-C):

Targeting container "api". …
Defaulting debug container name to debugger-bwhx9.
Warning: container debugger-bwhx9: container has runAsNonRoot and image will run as
root (pod: "api-646785c9b8-46xbl_notes(…)", container: debugger-bwhx9)
kubectl get "$POD" \
  -o jsonpath='{.status.ephemeralContainerStatuses[*].state.waiting.reason}'; echo
cat "$LAB/code/tag11/debug-user.yaml"

Expected:

CreateContainerConfigError
# debug-user.yaml – Ergänzung für kubectl debug --custom (Tag 11)
# busybox läuft sonst als root, runAsNonRoot im Pod verbietet das.
securityContext:
  runAsUser: 65532

The same trap as on Day 8: busybox runs as root. --custom extends the profile’s container definition with a fixed UID:

cp "$LAB/code/tag11/debug-user.yaml" .
kubectl debug -it "$POD" --image=busybox:1.37 --target=api --profile=restricted \
  --custom=debug-user.yaml -- sh

In the debug container:

id
ps
ls -l /proc/1/root/etc/notes/
cat /proc/1/root/etc/notes/api-key; echo
wget -qO- localhost:8080/version
ls /proc/1/root/
exit

Expected (abridged):

uid=65532 gid=0(root) groups=0(root)
PID   USER     TIME  COMMAND
    1 65532     0:02 /notes-api
   41 65532     0:00 sh
   55 65532     0:00 ps
total 0
lrwxrwxrwx    1 root     root            14 Oct 30 15:08 api-key -> ..data/api-key
lrwxrwxrwx    1 root     root            13 Oct 30 15:08 db-url -> ..data/db-url
lab-api-key-123
0.2.0
dev           notes-api     product_name  sys
etc           proc          product_uuid
Session ended, the ephemeral container will not be restarted but may be reattached …
  • --target=api shares the process namespace: PID 1 is /notes-api, and via /proc/1/root you see its file system including the Secret files.
  • localhost is the same Pod: the debug container shares the api’s network.
  • Ephemeral containers can’t be removed. They disappear with the Pod:
kubectl get "$POD" -o jsonpath='{.spec.ephemeralContainers[*].name}'; echo
kubectl rollout restart deploy/api && kubectl rollout status deploy/api

Expected (abridged):

debugger-bwhx9 debugger-r68kr
deployment.apps/api restarted
…
deployment "api" successfully rolled out

Security: Anyone allowed to run kubectl debug on a Pod can read its Secrets and processes. The permission behind it is patch on pods/ephemeralcontainers – grant it as sparingly as exec.

Save your progress? Sign up for free

Checkpoint

kubectl get gateway notes

Expected: PROGRAMMED True.

NAME    CLASS   ADDRESS      PROGRAMMED   AGE
notes   eg      172.18.0.3   True         28s
curl -s http://notes.localhost:30080/healthz; echo

Expected: notes answers via the Gateway.

ok
curl -s -i -H 'Host: falsch.localhost' localhost:30080/healthz | head -1

Expected: a wrong host gets a 404.

HTTP/1.1 404 Not Found
kubectl get networkpolicy

Expected: the four policies from the chart (more if the challenge policy is still there).

NAME                   POD-SELECTOR                 AGE
api-from-clients       app.kubernetes.io/name=api   30s
db-from-api            app.kubernetes.io/name=db    30s
default-deny-ingress   <none>                       30s
web-from-gateway       app.kubernetes.io/name=web   30s
kubectl exec deploy/web -- nc -z -w 3 db 5432

Expected: web can’t reach the database.

command terminated with exit code 1

Challenge

The Pod fremd from Drill 7 should be allowed to reach the api, but still not web. Change nothing in the chart or in the existing policies to do so. Prove both, take the permission back again and clean up fremd.

Hint: One additional policy is enough; namespaceSelector and podSelector in the same from entry mean and.

Solution

cat > k8s/api-from-fremd.yaml <<'EOF'
# k8s/api-from-fremd.yaml – Challenge Day 11 (delete again)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-fremd
  namespace: notes
spec:
  podSelector:
    matchLabels: {app.kubernetes.io/name: api}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: default}
          podSelector:
            matchLabels: {run: fremd}
      ports:
        - port: 8080
EOF
kubectl apply --dry-run=server -f k8s/api-from-fremd.yaml
kubectl apply -f k8s/api-from-fremd.yaml
kubectl -n default get pod fremd --show-labels
kubectl -n default exec fremd -- wget -qO- -T 3 http://api.notes:8080/version
kubectl -n default exec fremd -- wget -qO- -T 3 http://web.notes:3000/healthz
kubectl delete -f k8s/api-from-fremd.yaml && rm k8s/api-from-fremd.yaml
kubectl -n default exec fremd -- wget -qO- -T 3 http://api.notes:8080/version
kubectl -n default delete pod fremd

Expected (abridged):

networkpolicy.networking.k8s.io/api-from-fremd created (server dry run)
networkpolicy.networking.k8s.io/api-from-fremd created
NAME    READY   STATUS    RESTARTS   AGE   LABELS
fremd   1/1     Running   0          53s   run=fremd
0.2.0
wget: download timed out
command terminated with exit code 1
networkpolicy.networking.k8s.io "api-from-fremd" deleted from notes namespace
wget: download timed out
command terminated with exit code 1
pod "fremd" deleted from default namespace

kubectl run sets the label run=fremd. The new policy only extends the permissions of the api; for web, web-from-gateway still applies. With - namespaceSelector and - podSelector as two entries, every Pod from default or every Pod with run=fremd from notes would have been granted access.

Save your progress? Sign up for free

Enterprise Note

  • TLS belongs on the Gateway: an HTTPS listener with certificateRefs pointing to a Secret that cert-manager issues and renews automatically. More in “Gateway API hands-on”.
  • In front of it sits a real load balancer (cloud, MetalLB, kube-vip) instead of a NodePort; then externalTrafficPolicy: Local also makes sense, because it preserves the client IP.
  • Default deny for Egress too, with exceptions for DNS (kube-dns, port 53) and the database. If you want to enforce policies for all namespaces, use Kyverno – see “Kyverno & NetworkPolicies hands-on”.
  • Not every CNI enforces NetworkPolicies. Check it as in Drill 7 with a foreign Pod before you rely on it.
  • kubectl debug remains the exception: permissions on pods/ephemeralcontainers only for on-call staff and with an audit log; debug images from your own registry.