Day 11: Reachable from Outside, Sealed Off Inside
Goal: Make notes reachable through a Gateway at
http://notes.localhost:30080, limit the traffic in the namespace to what is necessary with NetworkPolicies, and look inside the api’s scratch container. · Duration: approx. 120 min · Prerequisite: Day 10
Briefing
- Gateway API separates roles: the platform provides a GatewayClass (which controller, which parameters), the app team creates the Gateway (listener: port, hostname) and the HTTPRoute (host, path → Service).
- The controller here is Envoy Gateway: for each
Gateway it starts an Envoy Pod plus a Service in the namespace
envoy-gateway-system. - A NetworkPolicy selects Pods by label. As soon as a policy selects a Pod, only what a policy allows is permitted for it.
kubectl debugattaches an ephemeral container with tools to a running Pod – even to one without a shell.
Mac: curl http://notes.localhost:30080
│ extraPortMappings (Day 0)
▼
Node :30080 ─► Service envoy-… (NodePort) ─► Pod envoy envoy-gateway-system
│ Gateway notes, HTTPRoute web
▼ Host: notes.localhost
Service web (ClusterIP) ─► web ─► api ─► db notes
Drill 1: Start of the Day and Envoy Gateway
source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag11 && cd ~/k8s-lab/tag11
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag10/notes/." .
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.3 \
-n envoy-gateway-system --create-namespace --wait | grep -E '^(NAME|STATUS):'
kubectl -n envoy-gateway-system get pods
kubectl get crd | grep -c gateway.networking.k8s.io
kubectl get crd gateways.gateway.networking.k8s.io \
-o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}'
echoExpected (abridged, names, IPs and times will differ on your machine):
Pulled: docker.io/envoyproxy/gateway-helm:v1.9.3
Digest: sha256:be1b8f16…
NAME: eg
STATUS: deployed
NAME READY STATUS RESTARTS AGE
eg-gateway-helm-certgen-xbb8q 0/1 Completed 0 42s
envoy-gateway-6776489b4b-rnzg5 1/1 Running 0 19s
10
v1.6.1
The chart ships the Gateway API CRDs (here bundle v1.6.1) and Envoy Gateway’s own CRDs. The certgen Job generates the controller’s certificates once. This takes just under a minute.
Save your progress? Sign up for free
Drill 2: GatewayClass – the Platform Side
The GatewayClass eg refers to an
EnvoyProxy object with parameters: the Envoy Service
should be a NodePort on 30080, because that is the only port kind
forwards to the Mac. First draft:
cat > k8s/gateway.yaml <<'EOF'
# k8s/gateway.yaml – first draft (Day 11, Drill 2; finished in Drill 4)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: nodeport
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
type: NodePort
patch:
type: StrategicMerge
value:
spec:
ports:
- port: 80
nodePort: 30080
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: nodeport
namespace: envoy-gateway-system
EOF
kubectl apply --dry-run=server -f k8s/gateway.yaml
kubectl apply -f k8s/gateway.yaml
kubectl get gatewayclassExpected:
envoyproxy.gateway.envoyproxy.io/nodeport created (server dry run)
gatewayclass.gateway.networking.k8s.io/eg created (server dry run)
envoyproxy.gateway.envoyproxy.io/nodeport created
gatewayclass.gateway.networking.k8s.io/eg created
NAME CONTROLLER ACCEPTED AGE
eg gateway.envoyproxy.io/gatewayclass-controller True 0s
ACCEPTED True means: the controller considers itself
responsible. There is no Envoy Pod yet – it is only created with the
first Gateway. Like namespace.yaml,
gateway.yaml belongs to the platform part in
k8s/, not in the chart.
Save your progress? Sign up for free
Drill 3: Chart 0.2.0 – Gateway and Route
The app team delivers the Gateway and HTTPRoute in the chart. Fetch the state from the companion code and look at the changes:
cp -R "$LAB/code/tag11/notes/chart/." chart/
grep -E '^version' chart/notes/Chart.yaml
sed -n '/^gateway:/,$p' chart/notes/values.yaml
grep -A1 '^ service:' chart/notes/values.yamlExpected:
version: 0.2.0
gateway:
enabled: true
className: eg
hostname: notes.localhost
networkPolicy:
enabled: true
gatewayNamespace: envoy-gateway-system
service:
type: ClusterIP # ab Tag 11 kommt der Verkehr über das Gateway
The new template (the NetworkPolicies from the same delivery follow in Drill 6):
# chart/notes/templates/gateway.yaml – state at end of Day 11
{{- if .Values.gateway.enabled }}
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: notes
labels:
{{- include "notes.labels" (dict "ctx" $ "name" "gateway") | nindent 4 }}
spec:
gatewayClassName: {{ .Values.gateway.className }}
listeners:
- name: http
protocol: HTTP
port: 80
hostname: {{ .Values.gateway.hostname | quote }}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: web
labels:
{{- include "notes.labels" (dict "ctx" $ "name" "web") | nindent 4 }}
spec:
parentRefs:
- name: notes
hostnames:
- {{ .Values.gateway.hostname | quote }}
rules:
- backendRefs:
- name: web
port: 3000
{{- end }}- The listener only accepts requests for
notes.localhoston port 80. Port 80 is the port of the Envoy Service; the NodePort 30080 comes from the EnvoyProxy patch. - The route attaches itself to the Gateway via
parentRefsand sends everything to the Serviceweb. Withoutmatchesit applies to every path. - web becomes
ClusterIP: from now on, the NodePort 30080 belongs to the Gateway.
helm lint chart/notes | tail -1
helm template notes chart/notes -n notes \
| kubectl apply --dry-run=server -f - 2>/dev/null | grep created
helm upgrade notes chart/notes -n notes --wait | tail -3
kubectl get svc/web gateway/notes httproute/webExpected (abridged):
1 chart(s) linted, 0 chart(s) failed
networkpolicy.networking.k8s.io/default-deny-ingress created (server dry run)
networkpolicy.networking.k8s.io/web-from-gateway created (server dry run)
networkpolicy.networking.k8s.io/api-from-clients created (server dry run)
networkpolicy.networking.k8s.io/db-from-api created (server dry run)
gateway.gateway.networking.k8s.io/notes created (server dry run)
httproute.gateway.networking.k8s.io/web created (server dry run)
pod/notes-test-api created (server dry run)
notes 0.2.0 ist im Namespace notes installiert.
Über das Gateway: http://notes.localhost:30080
Test: helm test notes -n notes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/web ClusterIP 10.96.196.80 <none> 3000/TCP 4m
NAME CLASS ADDRESS PROGRAMMED AGE
gateway.gateway.networking.k8s.io/notes eg 172.18.0.6 True 26s
NAME HOSTNAMES AGE
httproute.gateway.networking.k8s.io/web ["notes.localhost"] 26s
2>/dev/null hides harmless messages (missing
last-applied annotation on the Helm objects,
field is immutable for the hook Job as on Day 10). Because
the chart version in the labels changes, all Pods roll once; api may
restart once or twice along the way until db is ready again (Day 7).
Save your progress? Sign up for free
Drill 4: The externalTrafficPolicy
Trap
PROGRAMMED True – so let’s go:
curl -sS -m 5 http://notes.localhost:30080/healthz; echo "Exit: $?"Expected:
curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
Exit: 28
The connection is established, but nobody answers. Where does Envoy run, and how is its Service configured?
kubectl -n envoy-gateway-system get pods \
-o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName'
c='NAME:.metadata.name,TYPE:.spec.type'
kubectl -n envoy-gateway-system get svc \
-l gateway.envoyproxy.io/owning-gateway-name=notes \
-o custom-columns="$c,POLICY:.spec.externalTrafficPolicy"Expected (abridged):
NAME NODE
envoy-gateway-6776489b4b-rnzg5 lab-worker
envoy-notes-notes-3aa9e90e-646c5fd4cc-88ss8 lab-worker2
NAME TYPE POLICY
envoy-notes-notes-3aa9e90e NodePort Local
Envoy Gateway sets externalTrafficPolicy: Local: a node
only forwards traffic from outside to Pods on itself.
With kind, however, port 30080 always arrives at the control plane node,
and no Envoy runs there. With Cluster, every node may
forward to any Envoy Pod. Add two lines to the EnvoyProxy patch:
# k8s/gateway.yaml (excerpt) – state at end of Day 11
spec:
# The default is Local: then only a node on which Envoy runs answers.
externalTrafficPolicy: Cluster
ports:
- port: 80
nodePort: 30080The complete file is in
code/tag11/notes/k8s/gateway.yaml:
cp "$LAB/code/tag11/notes/k8s/gateway.yaml" k8s/
kubectl diff -f k8s/gateway.yaml | grep -E '^[-+] '
kubectl apply -f k8s/gateway.yaml
kubectl -n envoy-gateway-system get svc \
-l gateway.envoyproxy.io/owning-gateway-name=notes \
-o custom-columns='NAME:.metadata.name,POLICY:.spec.externalTrafficPolicy'
curl -sS -m 5 http://notes.localhost:30080/healthz; echo
curl -s http://notes.localhost:30080/api/notes | jq -c 'map(.text)'Expected (abridged, the notes come from Day 10):
- generation: 1
+ generation: 2
+ externalTrafficPolicy: Cluster
envoyproxy.gateway.envoyproxy.io/nodeport configured
gatewayclass.gateway.networking.k8s.io/eg unchanged
NAME POLICY
envoy-notes-notes-3aa9e90e Cluster
ok
["Postgres eins","Postgres zwei","Postgres drei","Kontrollpunkt 8"]
Envoy Gateway applies the change to the EnvoyProxy to the Service
immediately. The notes now come via Envoy → web → api → Postgres.
http://notes.localhost:30080 also works in the browser.
Tip: curl on the Mac resolves
*.localhostto 127.0.0.1 by itself, without/etc/hosts. If it doesn’t on your machine (older curl versions, other tools), set the Host header by hand:curl -H 'Host: notes.localhost' localhost:30080/…orcurl --resolve notes.localhost:30080:127.0.0.1 http://notes.localhost:30080/….
A test on the node itself
(docker exec lab-control-plane curl … 127.0.0.1:30080)
would have returned ok even before – kube-proxy treats
traffic from the node itself differently. That is why a test on the node
is deceptive: always check the path that the user takes too.
Save your progress? Sign up for free
Drill 5: Hostnames and 404
The Gateway decides based on the Host header:
curl -s -o /dev/null -w '%{http_code}\n' localhost:30080/healthz
curl -si -H 'Host: falsch.localhost' localhost:30080/healthz
curl -si -H 'Host: notes.localhost' localhost:30080/healthz | head -2
curl -s http://notes.localhost:30080/api/notes -H 'Content-Type: application/json' \
-d '{"text":"über das Gateway"}'; echoExpected (abridged):
404
HTTP/1.1 404 Not Found
date: Fri, 30 Oct 2026 15:09:03 GMT
content-length: 0
HTTP/1.1 200 OK
x-powered-by: Express
{"id":"68be9f2b-…","text":"über das Gateway"}
For localhost and falsch.localhost there is
neither a listener nor a route, so Envoy itself answers with an empty
404. The 200 with x-powered-by: Express comes from
notes-web. The state of the Gateway and the route is shown by their
conditions:
kubectl get gateway notes \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}'; echo
kubectl get httproute web \
-o jsonpath='{range .status.parents[0].conditions[*]}{.type}={.status} {end}'; echoExpected:
Accepted=True Programmed=True
Accepted=True ResolvedRefs=True
ResolvedRefs=False would mean: the Service in
backendRefs doesn’t exist or the port is wrong. That is the
first place you look when you get a 404 or 500.
Save your progress? Sign up for free
Drill 6: NetworkPolicy – Close Everything, Then Open Selectively
The upgrade from Drill 3 also created four NetworkPolicies. kindnet,
kind’s Pod network, enforces them. The template (excerpt, complete in
code/tag11/notes/chart/notes/templates/networkpolicy.yaml):
# chart/notes/templates/networkpolicy.yaml (excerpt) – state at end of Day 11
# 1. Deny all incoming traffic in the namespace …
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
spec:
podSelector: {}
policyTypes: [Ingress]
---
# 2. … web only from the Gateway
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-from-gateway
spec:
podSelector:
matchLabels: {app.kubernetes.io/name: web}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: {{ .Values.networkPolicy.gatewayNamespace }}
- podSelector:
matchLabels: {app.kubernetes.io/name: test}
ports:
- port: 3000podSelector: {}selects all Pods in the namespace; without aningressrule, all incoming traffic is thereby denied.- Policies add up: web may additionally be addressed from
envoy-gateway-systemand by the Helm test Pod. Two entries underfromare an or. - Following the same pattern:
api-from-clients(api only from web, worker, test, port- and
db-from-api(db only from api and migrate, port 5432).
- and
- Outgoing traffic (
Egress) stays open; so DNS keeps working.
kubectl get networkpolicy
kubectl describe networkpolicy api-from-clients | sed -n '/^Spec/,$p'Expected (abridged):
NAME POD-SELECTOR AGE
api-from-clients app.kubernetes.io/name=api 2m38s
db-from-api app.kubernetes.io/name=db 2m38s
default-deny-ingress <none> 2m38s
web-from-gateway app.kubernetes.io/name=web 2m38s
Spec:
PodSelector: app.kubernetes.io/name=api
Allowing ingress traffic:
To Port: 8080/TCP
From:
PodSelector: app.kubernetes.io/name in (test,web,worker)
Not affecting egress traffic
Policy Types: Ingress
Save your progress? Sign up for free
Drill 7: Proof with a Foreign Pod
A policy you haven’t tested is a guess. A Pod in the namespace
default plays the intruder:
kubectl -n default run fremd --image=busybox:1.37 --restart=Never -- sleep 3600
kubectl -n default wait --for=condition=Ready pod/fremd --timeout=120s
kubectl -n default exec fremd -- wget -qO- -T 3 http://api.notes:8080/version
kubectl -n default exec fremd -- wget -qO- -T 3 http://web.notes:3000/healthzExpected:
pod/fremd created
pod/fremd condition met
wget: download timed out
command terminated with exit code 1
wget: download timed out
command terminated with exit code 1
No Connection refused as with an empty Service (Day 3),
but a timeout: the packets are silently dropped. Now the allowed and
forbidden paths from the inside:
kubectl exec deploy/web -- wget -qO- -T 3 http://api:8080/version
kubectl exec deploy/web -- nc -z -w 3 db 5432; echo "web → db: Exit $?"
kubectl logs deploy/worker --tail 1
kubectl get pods
helm test notes -n notes | grep PhaseExpected (abridged):
0.2.0
command terminated with exit code 1
web → db: Exit 1
summary: 5 notes, 11 words
NAME READY STATUS RESTARTS AGE
api-646785c9b8-46xbl 1/1 Running 0 95s
api-646785c9b8-fs2l9 1/1 Running 2 (108s ago) 113s
api-646785c9b8-hrrmd 1/1 Running 0 91s
db-0 1/1 Running 0 110s
…
web-7955fff6b-wjv6q 1/1 Running 0 113s
worker-847b98cbbd-dn8fp 1/1 Running 0 112s
Phase: Succeeded
web may talk to the api, but not to the database; worker keeps
counting; all Pods stay Ready. The kubelet’s probes are
exempt from NetworkPolicies. You still need the Pod fremd
in the challenge.
Save your progress? Sign up for free
Drill 8: kubectl debug Under
restricted
notes-api is scratch: no shell, no ps (Day 1).
POD=$(kubectl get pods -l app.kubernetes.io/name=api -o name | head -1)
kubectl exec "$POD" -- shExpected (abridged):
error: Internal error occurred: … exec: "sh": executable file not found in $PATH
So a debug container with busybox that sees the processes of the
container api:
kubectl debug -it "$POD" --image=busybox:1.37 --target=api -- shExpected (abridged):
Targeting container "api". If you don't see processes from this container it may …
Defaulting debug container name to debugger-js86b.
Error from server (Forbidden): pods "api-646785c9b8-46xbl" is forbidden: violates
PodSecurity "restricted:latest": allowPrivilegeEscalation != false (…), unrestricted
capabilities (… must not include "SYS_PTRACE" in securityContext.capabilities.add)
The default profile general adds SYS_PTRACE
– which restricted forbids (Day 8). The profile
restricted sets the appropriate fields:
kubectl debug -it "$POD" --image=busybox:1.37 --target=api --profile=restricted -- shExpected (abridged; kubectl waits afterwards, abort with
Ctrl-C):
Targeting container "api". …
Defaulting debug container name to debugger-bwhx9.
Warning: container debugger-bwhx9: container has runAsNonRoot and image will run as
root (pod: "api-646785c9b8-46xbl_notes(…)", container: debugger-bwhx9)
kubectl get "$POD" \
-o jsonpath='{.status.ephemeralContainerStatuses[*].state.waiting.reason}'; echo
cat "$LAB/code/tag11/debug-user.yaml"Expected:
CreateContainerConfigError
# debug-user.yaml – Ergänzung für kubectl debug --custom (Tag 11)
# busybox läuft sonst als root, runAsNonRoot im Pod verbietet das.
securityContext:
runAsUser: 65532
The same trap as on Day 8: busybox runs as root.
--custom extends the profile’s container definition with a
fixed UID:
cp "$LAB/code/tag11/debug-user.yaml" .
kubectl debug -it "$POD" --image=busybox:1.37 --target=api --profile=restricted \
--custom=debug-user.yaml -- shIn the debug container:
id
ps
ls -l /proc/1/root/etc/notes/
cat /proc/1/root/etc/notes/api-key; echo
wget -qO- localhost:8080/version
ls /proc/1/root/
exitExpected (abridged):
uid=65532 gid=0(root) groups=0(root)
PID USER TIME COMMAND
1 65532 0:02 /notes-api
41 65532 0:00 sh
55 65532 0:00 ps
total 0
lrwxrwxrwx 1 root root 14 Oct 30 15:08 api-key -> ..data/api-key
lrwxrwxrwx 1 root root 13 Oct 30 15:08 db-url -> ..data/db-url
lab-api-key-123
0.2.0
dev notes-api product_name sys
etc proc product_uuid
Session ended, the ephemeral container will not be restarted but may be reattached …
--target=apishares the process namespace: PID 1 is/notes-api, and via/proc/1/rootyou see its file system including the Secret files.localhostis the same Pod: the debug container shares the api’s network.- Ephemeral containers can’t be removed. They disappear with the Pod:
kubectl get "$POD" -o jsonpath='{.spec.ephemeralContainers[*].name}'; echo
kubectl rollout restart deploy/api && kubectl rollout status deploy/apiExpected (abridged):
debugger-bwhx9 debugger-r68kr
deployment.apps/api restarted
…
deployment "api" successfully rolled out
Security: Anyone allowed to run
kubectl debugon a Pod can read its Secrets and processes. The permission behind it ispatchonpods/ephemeralcontainers– grant it as sparingly asexec.
Save your progress? Sign up for free
Checkpoint
kubectl get gateway notesExpected: PROGRAMMED True.
NAME CLASS ADDRESS PROGRAMMED AGE
notes eg 172.18.0.3 True 28s
curl -s http://notes.localhost:30080/healthz; echoExpected: notes answers via the Gateway.
ok
curl -s -i -H 'Host: falsch.localhost' localhost:30080/healthz | head -1Expected: a wrong host gets a 404.
HTTP/1.1 404 Not Found
kubectl get networkpolicyExpected: the four policies from the chart (more if the challenge policy is still there).
NAME POD-SELECTOR AGE
api-from-clients app.kubernetes.io/name=api 30s
db-from-api app.kubernetes.io/name=db 30s
default-deny-ingress <none> 30s
web-from-gateway app.kubernetes.io/name=web 30s
kubectl exec deploy/web -- nc -z -w 3 db 5432Expected: web can’t reach the database.
command terminated with exit code 1
Challenge
The Pod fremd from Drill 7 should be allowed to reach
the api, but still not web. Change nothing in the chart or in the
existing policies to do so. Prove both, take the permission back again
and clean up fremd.
Hint: One additional policy is enough;
namespaceSelector and podSelector in the
same from entry mean
and.
Solution
cat > k8s/api-from-fremd.yaml <<'EOF'
# k8s/api-from-fremd.yaml – Challenge Day 11 (delete again)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-from-fremd
namespace: notes
spec:
podSelector:
matchLabels: {app.kubernetes.io/name: api}
ingress:
- from:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: default}
podSelector:
matchLabels: {run: fremd}
ports:
- port: 8080
EOF
kubectl apply --dry-run=server -f k8s/api-from-fremd.yaml
kubectl apply -f k8s/api-from-fremd.yaml
kubectl -n default get pod fremd --show-labels
kubectl -n default exec fremd -- wget -qO- -T 3 http://api.notes:8080/version
kubectl -n default exec fremd -- wget -qO- -T 3 http://web.notes:3000/healthz
kubectl delete -f k8s/api-from-fremd.yaml && rm k8s/api-from-fremd.yaml
kubectl -n default exec fremd -- wget -qO- -T 3 http://api.notes:8080/version
kubectl -n default delete pod fremdExpected (abridged):
networkpolicy.networking.k8s.io/api-from-fremd created (server dry run)
networkpolicy.networking.k8s.io/api-from-fremd created
NAME READY STATUS RESTARTS AGE LABELS
fremd 1/1 Running 0 53s run=fremd
0.2.0
wget: download timed out
command terminated with exit code 1
networkpolicy.networking.k8s.io "api-from-fremd" deleted from notes namespace
wget: download timed out
command terminated with exit code 1
pod "fremd" deleted from default namespace
kubectl run sets the label run=fremd. The
new policy only extends the permissions of the api; for web,
web-from-gateway still applies. With
- namespaceSelector and - podSelector as two
entries, every Pod from default or every
Pod with run=fremd from notes would have been
granted access.
Save your progress? Sign up for free
Enterprise Note
- TLS belongs on the Gateway: an HTTPS listener with
certificateRefspointing to a Secret that cert-manager issues and renews automatically. More in “Gateway API hands-on”. - In front of it sits a real load balancer (cloud, MetalLB, kube-vip)
instead of a NodePort; then
externalTrafficPolicy: Localalso makes sense, because it preserves the client IP. - Default deny for
Egresstoo, with exceptions for DNS (kube-dns, port 53) and the database. If you want to enforce policies for all namespaces, use Kyverno – see “Kyverno & NetworkPolicies hands-on”. - Not every CNI enforces NetworkPolicies. Check it as in Drill 7 with a foreign Pod before you rely on it.
kubectl debugremains the exception: permissions onpods/ephemeralcontainersonly for on-call staff and with an audit log; debug images from your own registry.