Day 11: Egress
Goal: You see which destinations outside the cluster your services call, block outbound traffic by default, and allow individual destinations via TLS and HTTP routes. · Duration: approx. 60 min · Prerequisite: Day 10
Briefing
- Without configuration, the proxy lets all outbound traffic through.
In the metrics this shows up as
egress-fallback; for HTTPS you only see IP addresses. - An EgressNetwork (
policy.linkerd.io) describes “everything outside the cluster” for the clients in its namespace, withtrafficPolicy: AlloworDeny. - A TLSRoute, HTTPRoute or TCPRoute with the EgressNetwork as its parent allows individual destinations – by SNI hostname or by HTTP host, path and method.
- Egress in the mesh is visibility and a guardrail, not a firewall: a Pod without a proxy bypasses it.
client ─> proxy ─┬─ TLSRoute example.com ──────> EgressNetwork internet ─> Internet
├─ HTTPRoute GET example.com ─> EgressNetwork internet ─> Internet
├─ HTTPRoute /version ────────> Service notes-api (in cluster)
└─ everything else ──> Deny: 403 or connection reset
Drill 1: Where Does the Traffic Go?
Under /upstream, notes-api calls the
address from NOTES_UPSTREAM. Add a client Pod for your own
experiments. The annotation
config.linkerd.io/proxy-metrics-hostname-labels writes the
destination name into the metrics – off by default, because every name
is a new time series.
mkdir -p ~/linkerd/tag11 && cd ~/linkerd/tag11
kubectl create namespace tag11
kubectl annotate namespace tag11 linkerd.io/inject=enabled
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: notes-api, namespace: tag11}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: client, namespace: tag11}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: tag11}
spec:
replicas: 1
selector: {matchLabels: {app: notes-api}}
template:
metadata:
labels: {app: notes-api}
annotations:
config.linkerd.io/proxy-metrics-hostname-labels: "true"
spec:
serviceAccountName: notes-api
containers:
- name: notes-api
image: ghcr.io/thomaszachmann/notes-api:0.7.1
env:
- {name: NOTES_API_KEY, value: lab-key-123}
- {name: NOTES_UPSTREAM, value: https://example.com}
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag11}
spec:
selector: {app: notes-api}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: client, namespace: tag11}
spec:
replicas: 1
selector: {matchLabels: {app: client}}
template:
metadata:
labels: {app: client}
annotations:
config.linkerd.io/proxy-metrics-hostname-labels: "true"
spec:
serviceAccountName: client
terminationGracePeriodSeconds: 2
containers:
- name: client
image: docker.io/curlimages/curl:8.23.0
command: [sleep, infinity]
EOF
kubectl apply -f app.yaml
kubectl -n tag11 rollout status deploy --timeout=120s
C="kubectl -n tag11 exec deploy/client -c client --"
for u in https://example.com http://example.com https://linkerd.io; do
$C curl -s -o /dev/null -w "$u %{http_code}\n" $u
done
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echoExpected (abridged):
https://example.com 200
http://example.com 200
https://linkerd.io 200
{"status":200,"url":"https://example.com"}
What did the client proxy see? The metrics are long;
this line extracts kind, route, destination, status and counter. You
will need it often today:
M() { linkerd diagnostics proxy-metrics -n tag11 deploy/client \
| grep -E '^outbound_(http_route_request_statuses|t(cp|ls)_route_open)_total' \
| sed -E 's/^outbound_([a-z]+)_.*route_name="([^"]*)",[a-z_]+="([^"]*)"/ \1 \2 \3 /' \
| sed -E 's/,http_status="([0-9]*)".*\}/\1/; s/[^ ]*\}//'; }
M | grep -v notes-apiExpected (abridged):
http egress-fallback example.com 200 1
tcp egress-fallback 172.66.147.243 1
tcp egress-fallback 188.114.96.0 1
For HTTP you see the hostname and status. For the proxy, HTTPS is just a TCP connection to an IP – the metric does not reveal which service is behind it.
Save your progress? Sign up for free
Drill 2: An EgressNetwork
cat > egress.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata: {name: internet, namespace: tag11}
spec:
trafficPolicy: Allow
EOF
kubectl apply -f egress.yaml
sleep 5
kubectl -n tag11 get egressnetwork internet \
-o jsonpath='{.status.conditions[0].reason}{"\n"}'
for u in https://example.com http://example.com https://linkerd.io; do
$C curl -s -o /dev/null -w "$u %{http_code}\n" $u
done
M | grep egress-allowExpected (abridged):
egressnetwork.policy.linkerd.io/internet created
Accepted
https://example.com 200
http://example.com 200
https://linkerd.io 200
http http-egress-allow example.com 200 1
tcp tcp-egress-allow 104.20.23.154 1
tcp tcp-egress-allow 172.66.147.243 1
Without networks, the EgressNetwork covers everything
outside the clusterNetworks from the Linkerd configuration.
The behavior does not change yet, but the traffic now has a parent:
http-egress-allow and tcp-egress-allow are the
built-in default routes. No restart needed – the proxy fetches the
policy live.
Tip: An EgressNetwork only applies to clients in its own namespace. One in the namespace
linkerd-egress(egress.globalEgressNetworkNamespace) applies to all. You deliberately do not create one in the lab – it would also affectnotes.
Save your progress? Sign up for free
Drill 3: Blocking Everything
kubectl -n tag11 patch egressnetwork internet --type=merge \
-p '{"spec":{"trafficPolicy":"Deny"}}'
sleep 5
for u in https://example.com http://example.com https://linkerd.io; do
$C curl -s -o /dev/null -w "$u %{http_code}\n" $u; echo "exit $?"
done
M | grep egress-deny
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo
kubectl -n tag11 rollout restart deploy/notes-api
kubectl -n tag11 rollout status deploy/notes-api
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echoExpected (abridged):
https://example.com 000
command terminated with exit code 35
exit 35
http://example.com 403
exit 0
https://linkerd.io 000
command terminated with exit code 35
exit 35
http http-egress-deny example.com 403 1
tcp tcp-egress-deny 104.20.23.154 1
tcp tcp-egress-deny 188.114.97.4 1
{"status":200,"url":"https://example.com"}
deployment "notes-api" successfully rolled out
{"error":"Get \"https://example.com\": read tcp 10.0.1.67:47058-\u003e104.20.23.154:443:
read: connection reset by peer"}
For HTTP, the proxy returns a 403. It cannot answer
HTTPS, so it drops the connection – curl reports exit code
35 (TLS handshake failed).
But notes-api still gets through! The policy applies to
new connections. The Go HTTP client in
notes-api keeps its connection to example.com
from Drill 1 open (keep-alive), and that connection persists. Only after
the restart does the block take effect. After a policy change, always
test with fresh connections – and in production, expect a block to take
full effect only after minutes or a rollout.
Save your progress? Sign up for free
Drill 4: Allowing a Destination via TLSRoute
With HTTPS, the proxy still has the hostname from the TLS handshake (SNI):
cat > tls-example.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata: {name: example-com, namespace: tag11}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 443}
hostnames: [example.com]
rules:
- backendRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
EOF
kubectl apply -f tls-example.yaml
sleep 5
kubectl -n tag11 get tlsroute example-com \
-o jsonpath='{.status.parents[*].controllerName}{"\n"}'
for u in https://example.com https://www.example.com https://linkerd.io; do
$C curl -s -o /dev/null -w "$u %{http_code}\n" $u
done
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo
M | grep '^ tls'Expected (abridged):
tlsroute.gateway.networking.k8s.io/example-com created
linkerd.io/policy-controller
https://example.com 200
https://www.example.com 000
command terminated with exit code 35
https://linkerd.io 000
command terminated with exit code 35
{"status":200,"url":"https://example.com"}
tls tls-egress-deny linkerd.io 1
tls tls-egress-deny www.example.com 1
tls example-com example.com 1
A backend reference to the EgressNetwork itself means “pass through
to the original destination”. As soon as a TLSRoute exists, the proxy
inspects the handshake on port 443 – tcp with an IP becomes
tls with a hostname, even for blocked destinations. Only
Linkerd accepted the route: Cilium cannot do anything with an
EgressNetwork as parent.
Save your progress? Sign up for free
Drill 5: Finer Control over HTTP
With plaintext HTTP, the proxy sees everything: host, method, path.
Only GET to example.com is to be allowed; you
redirect /version to notes-api in the cluster
– for example to replace an external API during testing:
cat > http-example.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: example-http, namespace: tag11}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 80}
hostnames: [example.com]
rules:
- matches:
- {method: GET, path: {type: PathPrefix, value: /}}
backendRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
- matches:
- {path: {type: PathPrefix, value: /version}}
backendRefs:
- {name: notes-api, port: 80}
EOF
kubectl apply -f http-example.yaml
sleep 5
$C curl -s -o /dev/null -w 'GET %{http_code}\n' http://example.com/
$C curl -s -o /dev/null -w 'POST %{http_code}\n' -X POST http://example.com/
$C curl -s -o /dev/null -w 'neverssl %{http_code}\n' http://neverssl.com/
$C curl -s -w ' %{http_code}\n' http://example.com/version
M | grep -E 'example-http|neverssl'Expected (abridged):
GET 200
POST 403
neverssl 403
0.7.1
200
http http-egress-deny neverssl.com 403 1
http example-http example.com 200 2
/version matches both rules; the longer path wins. The
response 0.7.1 comes from notes-api in the
cluster, even though curl called example.com.
The blocked POST is counted under
http-egress-deny with hostname example.com –
M | grep http-egress-deny shows it.
Save your progress? Sign up for free
Drill 6: A Second Destination for the App
From now on, notes-api is to query
https://linkerd.io:
kubectl -n tag11 set env deploy/notes-api NOTES_UPSTREAM=https://linkerd.io
kubectl -n tag11 rollout status deploy/notes-api
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo
kubectl -n tag11 patch tlsroute example-com --type=json \
-p '[{"op":"add","path":"/spec/hostnames/-","value":"linkerd.io"}]'
sleep 5
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echoExpected (abridged):
deployment "notes-api" successfully rolled out
{"error":"Get \"https://linkerd.io\": EOF"}
tlsroute.gateway.networking.k8s.io/example-com patched
{"status":200,"url":"https://linkerd.io"}
A new destination is a change to the route, not to the app. In
production it goes into the same pull request as the new
NOTES_UPSTREAM. To give Prometheus something to count,
generate some load and then query the destinations per service:
for i in $(seq 30); do
$C curl -s -o /dev/null -H 'X-API-Key: lab-key-123' http://notes-api/upstream
$C curl -s -o /dev/null https://www.example.com
sleep 2
done
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
Q='sum by (deployment, route_name, hostname)
(outbound_tls_route_open_total{namespace="tag11"})'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r \
'.data.result[] | [.metric.deployment, .metric.route_name, .metric.hostname,
(.value[1] | tonumber | round | tostring)] | join(" ")'Expected (numbers vary):
client tls-egress-deny www.example.com 30
client example-com example.com 1
client tls-egress-deny linkerd.io 1
notes-api tls-egress-deny linkerd.io 1
notes-api example-com linkerd.io 1
The metric counts connections, not requests:
client opens a new one for every curl (30),
while notes-api sent 30 requests over a single keep-alive
connection (1) – the same cause as the pitfall in Drill 3. This is the
egress overview for the whole namespace: who, where to, allowed or
blocked. An alert on route_name=~".*-egress-deny" reports
apps that try to reach a destination that has not been allowed. You will
need the port-forward again shortly.
Save your progress? Sign up for free
Drill 7: The Limit
kubectl -n tag11 run ohne --image=curlimages/curl:8.23.0 --restart=Never \
--annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl -n tag11 wait --for=condition=Ready pod/ohne --timeout=60s
kubectl -n tag11 exec ohne -- curl -s -o /dev/null -w '%{http_code}\n' \
https://www.example.comExpected:
pod/ohne created
pod/ohne condition met
200
No proxy, no rule. The same applies to Pods with
hostNetwork or a container with NET_ADMIN that
changes the iptables rules. The Linkerd docs say so explicitly: egress
control in the mesh is not a security guarantee. You draw a hard
boundary additionally in the network – with Cilium, for example, a
CiliumNetworkPolicy with toFQDNs, which you do
not touch in this lab, however.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all routes in
tag11attached to the EgressNetwork. Target time: 30 s - Switch
internettoAllowand back toDeny. Target time: 45 s - Show which hostnames were recently blocked for
client. Target time: 20 s - Check whether
notes-apiwrites hostnames into its metrics. Target time: 30 s
Solution
kubectl -n tag11 get httproute,tlsroute
kubectl -n tag11 patch egressnetwork internet --type=merge \
-p '{"spec":{"trafficPolicy":"Allow"}}'
kubectl -n tag11 patch egressnetwork internet --type=merge \
-p '{"spec":{"trafficPolicy":"Deny"}}'
M | grep egress-deny
kubectl -n tag11 get deploy notes-api \
-o jsonpath='{.spec.template.metadata.annotations}'Expected (excerpt for 1 and 4):
NAME HOSTNAMES AGE
httproute.gateway.networking.k8s.io/example-http ["example.com"] 6m
NAME AGE
tlsroute.gateway.networking.k8s.io/example-com 9m
{"config.linkerd.io/proxy-metrics-hostname-labels":"true"}
Save your progress? Sign up for free
Checkpoint
$C curl -s -o /dev/null -w '%{http_code}\n' https://example.com
$C curl -s -o /dev/null -w '%{http_code}\n' https://www.example.comExpected: 200, then 000
with exit code 35.
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echoExpected:
{"status":200,"url":"https://linkerd.io"}
kubectl -n tag11 get egressnetwork internet -o jsonpath='{.spec.trafficPolicy}{"\n"}'Expected: Deny
Challenge
Allow all subdomains of example.com without listing them
individually. Check whether example.com itself would still
be reachable through this entry afterwards.
Hint: hostnames in Gateway API routes
support leading wildcards.
Solution
kubectl -n tag11 patch tlsroute example-com --type=json \
-p '[{"op":"add","path":"/spec/hostnames/-","value":"*.example.com"}]'
sleep 5
for u in https://example.com https://www.example.com https://linkerd.io; do
$C curl -s -o /dev/null -w "$u %{http_code}\n" $u
done
kubectl -n tag11 patch tlsroute example-com --type=json \
-p '[{"op":"remove","path":"/spec/hostnames/0"}]'
sleep 5
$C curl -s -o /dev/null -w '%{http_code}\n' https://example.com
kill %1Expected (abridged):
tlsroute.gateway.networking.k8s.io/example-com patched
https://example.com 200
https://www.example.com 200
https://linkerd.io 200
tlsroute.gateway.networking.k8s.io/example-com patched
000
command terminated with exit code 35
After removing example.com (index 0), only
*.example.com remains – and it does not match
example.com itself. If you want both, you need both
entries.
Save your progress? Sign up for free
Cleanup
kubectl delete namespace tag11Interview Note
- How do you see which external destinations a service calls?
With an EgressNetwork (even set to
Allow) and hostname labels: the proxy metrics show the route, the SNI or HTTP hostname, and the status. - How do you block egress with Linkerd?
trafficPolicy: Deny, and allow destinations via TLS, HTTP or TCP routes with the EgressNetwork as parent and backend. - Is that a security boundary? No – Pods without a proxy bypass it. For hard boundaries, add network policies or an egress gateway.
Enterprise Note
- Observe first, then block: run
Allowwith hostname labels for weeks, build the allowlist from that, then switch toDeny. - Hostname labels increase cardinality – enable them only where you need them, or aggregate them with a recording rule.
- A global EgressNetwork in the
linkerd-egressnamespace sets a cluster-wide baseline rule; namespaces add their own allowances. - SNI allowances only check the name in the handshake. Against domain fronting, only a network policy on destination IPs or a proxy that breaks open TLS helps.
- Egress routes belong in the app’s repo: a new destination is a code change with review.