Skip to content
Hands-on IT IT Drills

Day 11: Egress

Goal: You see which destinations outside the cluster your services call, block outbound traffic by default, and allow individual destinations via TLS and HTTP routes. · Duration: approx. 60 min · Prerequisite: Day 10

Briefing

  • Without configuration, the proxy lets all outbound traffic through. In the metrics this shows up as egress-fallback; for HTTPS you only see IP addresses.
  • An EgressNetwork (policy.linkerd.io) describes “everything outside the cluster” for the clients in its namespace, with trafficPolicy: Allow or Deny.
  • A TLSRoute, HTTPRoute or TCPRoute with the EgressNetwork as its parent allows individual destinations – by SNI hostname or by HTTP host, path and method.
  • Egress in the mesh is visibility and a guardrail, not a firewall: a Pod without a proxy bypasses it.
 client ─> proxy ─┬─ TLSRoute example.com ──────> EgressNetwork internet ─> Internet
                  ├─ HTTPRoute GET example.com ─> EgressNetwork internet ─> Internet
                  ├─ HTTPRoute /version ────────> Service notes-api (in cluster)
                  └─ everything else ──> Deny: 403 or connection reset

Drill 1: Where Does the Traffic Go?

Under /upstream, notes-api calls the address from NOTES_UPSTREAM. Add a client Pod for your own experiments. The annotation config.linkerd.io/proxy-metrics-hostname-labels writes the destination name into the metrics – off by default, because every name is a new time series.

mkdir -p ~/linkerd/tag11 && cd ~/linkerd/tag11
kubectl create namespace tag11
kubectl annotate namespace tag11 linkerd.io/inject=enabled
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: notes-api, namespace: tag11}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: client, namespace: tag11}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: tag11}
spec:
  replicas: 1
  selector: {matchLabels: {app: notes-api}}
  template:
    metadata:
      labels: {app: notes-api}
      annotations:
        config.linkerd.io/proxy-metrics-hostname-labels: "true"
    spec:
      serviceAccountName: notes-api
      containers:
      - name: notes-api
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        env:
        - {name: NOTES_API_KEY, value: lab-key-123}
        - {name: NOTES_UPSTREAM, value: https://example.com}
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag11}
spec:
  selector: {app: notes-api}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: client, namespace: tag11}
spec:
  replicas: 1
  selector: {matchLabels: {app: client}}
  template:
    metadata:
      labels: {app: client}
      annotations:
        config.linkerd.io/proxy-metrics-hostname-labels: "true"
    spec:
      serviceAccountName: client
      terminationGracePeriodSeconds: 2
      containers:
      - name: client
        image: docker.io/curlimages/curl:8.23.0
        command: [sleep, infinity]
EOF
kubectl apply -f app.yaml
kubectl -n tag11 rollout status deploy --timeout=120s
C="kubectl -n tag11 exec deploy/client -c client --"
for u in https://example.com http://example.com https://linkerd.io; do
  $C curl -s -o /dev/null -w "$u %{http_code}\n" $u
done
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo

Expected (abridged):

https://example.com 200
http://example.com 200
https://linkerd.io 200
{"status":200,"url":"https://example.com"}

What did the client proxy see? The metrics are long; this line extracts kind, route, destination, status and counter. You will need it often today:

M() { linkerd diagnostics proxy-metrics -n tag11 deploy/client \
  | grep -E '^outbound_(http_route_request_statuses|t(cp|ls)_route_open)_total' \
  | sed -E 's/^outbound_([a-z]+)_.*route_name="([^"]*)",[a-z_]+="([^"]*)"/ \1 \2 \3 /' \
  | sed -E 's/,http_status="([0-9]*)".*\}/\1/; s/[^ ]*\}//'; }
M | grep -v notes-api

Expected (abridged):

 http egress-fallback example.com 200 1
 tcp egress-fallback 172.66.147.243  1
 tcp egress-fallback 188.114.96.0  1

For HTTP you see the hostname and status. For the proxy, HTTPS is just a TCP connection to an IP – the metric does not reveal which service is behind it.

Save your progress? Sign up for free

Drill 2: An EgressNetwork

cat > egress.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata: {name: internet, namespace: tag11}
spec:
  trafficPolicy: Allow
EOF
kubectl apply -f egress.yaml
sleep 5
kubectl -n tag11 get egressnetwork internet \
  -o jsonpath='{.status.conditions[0].reason}{"\n"}'
for u in https://example.com http://example.com https://linkerd.io; do
  $C curl -s -o /dev/null -w "$u %{http_code}\n" $u
done
M | grep egress-allow

Expected (abridged):

egressnetwork.policy.linkerd.io/internet created
Accepted
https://example.com 200
http://example.com 200
https://linkerd.io 200
 http http-egress-allow example.com 200 1
 tcp tcp-egress-allow 104.20.23.154  1
 tcp tcp-egress-allow 172.66.147.243  1

Without networks, the EgressNetwork covers everything outside the clusterNetworks from the Linkerd configuration. The behavior does not change yet, but the traffic now has a parent: http-egress-allow and tcp-egress-allow are the built-in default routes. No restart needed – the proxy fetches the policy live.

Tip: An EgressNetwork only applies to clients in its own namespace. One in the namespace linkerd-egress (egress.globalEgressNetworkNamespace) applies to all. You deliberately do not create one in the lab – it would also affect notes.

Save your progress? Sign up for free

Drill 3: Blocking Everything

kubectl -n tag11 patch egressnetwork internet --type=merge \
  -p '{"spec":{"trafficPolicy":"Deny"}}'
sleep 5
for u in https://example.com http://example.com https://linkerd.io; do
  $C curl -s -o /dev/null -w "$u %{http_code}\n" $u; echo "exit $?"
done
M | grep egress-deny
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo
kubectl -n tag11 rollout restart deploy/notes-api
kubectl -n tag11 rollout status deploy/notes-api
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo

Expected (abridged):

https://example.com 000
command terminated with exit code 35
exit 35
http://example.com 403
exit 0
https://linkerd.io 000
command terminated with exit code 35
exit 35
 http http-egress-deny example.com 403 1
 tcp tcp-egress-deny 104.20.23.154  1
 tcp tcp-egress-deny 188.114.97.4  1
{"status":200,"url":"https://example.com"}
deployment "notes-api" successfully rolled out
{"error":"Get \"https://example.com\": read tcp 10.0.1.67:47058-\u003e104.20.23.154:443:
read: connection reset by peer"}

For HTTP, the proxy returns a 403. It cannot answer HTTPS, so it drops the connection – curl reports exit code 35 (TLS handshake failed).

But notes-api still gets through! The policy applies to new connections. The Go HTTP client in notes-api keeps its connection to example.com from Drill 1 open (keep-alive), and that connection persists. Only after the restart does the block take effect. After a policy change, always test with fresh connections – and in production, expect a block to take full effect only after minutes or a rollout.

Save your progress? Sign up for free

Drill 4: Allowing a Destination via TLSRoute

With HTTPS, the proxy still has the hostname from the TLS handshake (SNI):

cat > tls-example.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata: {name: example-com, namespace: tag11}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 443}
  hostnames: [example.com]
  rules:
  - backendRefs:
    - {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
EOF
kubectl apply -f tls-example.yaml
sleep 5
kubectl -n tag11 get tlsroute example-com \
  -o jsonpath='{.status.parents[*].controllerName}{"\n"}'
for u in https://example.com https://www.example.com https://linkerd.io; do
  $C curl -s -o /dev/null -w "$u %{http_code}\n" $u
done
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo
M | grep '^ tls'

Expected (abridged):

tlsroute.gateway.networking.k8s.io/example-com created
linkerd.io/policy-controller
https://example.com 200
https://www.example.com 000
command terminated with exit code 35
https://linkerd.io 000
command terminated with exit code 35
{"status":200,"url":"https://example.com"}
 tls tls-egress-deny linkerd.io  1
 tls tls-egress-deny www.example.com  1
 tls example-com example.com  1

A backend reference to the EgressNetwork itself means “pass through to the original destination”. As soon as a TLSRoute exists, the proxy inspects the handshake on port 443 – tcp with an IP becomes tls with a hostname, even for blocked destinations. Only Linkerd accepted the route: Cilium cannot do anything with an EgressNetwork as parent.

Save your progress? Sign up for free

Drill 5: Finer Control over HTTP

With plaintext HTTP, the proxy sees everything: host, method, path. Only GET to example.com is to be allowed; you redirect /version to notes-api in the cluster – for example to replace an external API during testing:

cat > http-example.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: example-http, namespace: tag11}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 80}
  hostnames: [example.com]
  rules:
  - matches:
    - {method: GET, path: {type: PathPrefix, value: /}}
    backendRefs:
    - {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
  - matches:
    - {path: {type: PathPrefix, value: /version}}
    backendRefs:
    - {name: notes-api, port: 80}
EOF
kubectl apply -f http-example.yaml
sleep 5
$C curl -s -o /dev/null -w 'GET  %{http_code}\n' http://example.com/
$C curl -s -o /dev/null -w 'POST %{http_code}\n' -X POST http://example.com/
$C curl -s -o /dev/null -w 'neverssl %{http_code}\n' http://neverssl.com/
$C curl -s -w ' %{http_code}\n' http://example.com/version
M | grep -E 'example-http|neverssl'

Expected (abridged):

GET  200
POST 403
neverssl 403
0.7.1
 200
 http http-egress-deny neverssl.com 403 1
 http example-http example.com 200 2

/version matches both rules; the longer path wins. The response 0.7.1 comes from notes-api in the cluster, even though curl called example.com. The blocked POST is counted under http-egress-deny with hostname example.com – M | grep http-egress-deny shows it.

Save your progress? Sign up for free

Drill 6: A Second Destination for the App

From now on, notes-api is to query https://linkerd.io:

kubectl -n tag11 set env deploy/notes-api NOTES_UPSTREAM=https://linkerd.io
kubectl -n tag11 rollout status deploy/notes-api
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo
kubectl -n tag11 patch tlsroute example-com --type=json \
  -p '[{"op":"add","path":"/spec/hostnames/-","value":"linkerd.io"}]'
sleep 5
$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo

Expected (abridged):

deployment "notes-api" successfully rolled out
{"error":"Get \"https://linkerd.io\": EOF"}
tlsroute.gateway.networking.k8s.io/example-com patched
{"status":200,"url":"https://linkerd.io"}

A new destination is a change to the route, not to the app. In production it goes into the same pull request as the new NOTES_UPSTREAM. To give Prometheus something to count, generate some load and then query the destinations per service:

for i in $(seq 30); do
  $C curl -s -o /dev/null -H 'X-API-Key: lab-key-123' http://notes-api/upstream
  $C curl -s -o /dev/null https://www.example.com
  sleep 2
done
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
Q='sum by (deployment, route_name, hostname)
 (outbound_tls_route_open_total{namespace="tag11"})'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r \
  '.data.result[] | [.metric.deployment, .metric.route_name, .metric.hostname,
   (.value[1] | tonumber | round | tostring)] | join(" ")'

Expected (numbers vary):

client tls-egress-deny www.example.com 30
client example-com example.com 1
client tls-egress-deny linkerd.io 1
notes-api tls-egress-deny linkerd.io 1
notes-api example-com linkerd.io 1

The metric counts connections, not requests: client opens a new one for every curl (30), while notes-api sent 30 requests over a single keep-alive connection (1) – the same cause as the pitfall in Drill 3. This is the egress overview for the whole namespace: who, where to, allowed or blocked. An alert on route_name=~".*-egress-deny" reports apps that try to reach a destination that has not been allowed. You will need the port-forward again shortly.

Save your progress? Sign up for free

Drill 7: The Limit

kubectl -n tag11 run ohne --image=curlimages/curl:8.23.0 --restart=Never \
  --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl -n tag11 wait --for=condition=Ready pod/ohne --timeout=60s
kubectl -n tag11 exec ohne -- curl -s -o /dev/null -w '%{http_code}\n' \
  https://www.example.com

Expected:

pod/ohne created
pod/ohne condition met
200

No proxy, no rule. The same applies to Pods with hostNetwork or a container with NET_ADMIN that changes the iptables rules. The Linkerd docs say so explicitly: egress control in the mesh is not a security guarantee. You draw a hard boundary additionally in the network – with Cilium, for example, a CiliumNetworkPolicy with toFQDNs, which you do not touch in this lab, however.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all routes in tag11 attached to the EgressNetwork. Target time: 30 s
  2. Switch internet to Allow and back to Deny. Target time: 45 s
  3. Show which hostnames were recently blocked for client. Target time: 20 s
  4. Check whether notes-api writes hostnames into its metrics. Target time: 30 s

Solution

kubectl -n tag11 get httproute,tlsroute
kubectl -n tag11 patch egressnetwork internet --type=merge \
  -p '{"spec":{"trafficPolicy":"Allow"}}'
kubectl -n tag11 patch egressnetwork internet --type=merge \
  -p '{"spec":{"trafficPolicy":"Deny"}}'
M | grep egress-deny
kubectl -n tag11 get deploy notes-api \
  -o jsonpath='{.spec.template.metadata.annotations}'

Expected (excerpt for 1 and 4):

NAME                                               HOSTNAMES         AGE
httproute.gateway.networking.k8s.io/example-http   ["example.com"]   6m

NAME                                             AGE
tlsroute.gateway.networking.k8s.io/example-com   9m
{"config.linkerd.io/proxy-metrics-hostname-labels":"true"}

Save your progress? Sign up for free

Checkpoint

$C curl -s -o /dev/null -w '%{http_code}\n' https://example.com
$C curl -s -o /dev/null -w '%{http_code}\n' https://www.example.com

Expected: 200, then 000 with exit code 35.

$C curl -s -H 'X-API-Key: lab-key-123' http://notes-api/upstream; echo

Expected: {"status":200,"url":"https://linkerd.io"}

kubectl -n tag11 get egressnetwork internet -o jsonpath='{.spec.trafficPolicy}{"\n"}'

Expected: Deny

Challenge

Allow all subdomains of example.com without listing them individually. Check whether example.com itself would still be reachable through this entry afterwards.

Hint: hostnames in Gateway API routes support leading wildcards.

Solution

kubectl -n tag11 patch tlsroute example-com --type=json \
  -p '[{"op":"add","path":"/spec/hostnames/-","value":"*.example.com"}]'
sleep 5
for u in https://example.com https://www.example.com https://linkerd.io; do
  $C curl -s -o /dev/null -w "$u %{http_code}\n" $u
done
kubectl -n tag11 patch tlsroute example-com --type=json \
  -p '[{"op":"remove","path":"/spec/hostnames/0"}]'
sleep 5
$C curl -s -o /dev/null -w '%{http_code}\n' https://example.com
kill %1

Expected (abridged):

tlsroute.gateway.networking.k8s.io/example-com patched
https://example.com 200
https://www.example.com 200
https://linkerd.io 200
tlsroute.gateway.networking.k8s.io/example-com patched
000
command terminated with exit code 35

After removing example.com (index 0), only *.example.com remains – and it does not match example.com itself. If you want both, you need both entries.

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag11

Interview Note

  • How do you see which external destinations a service calls? With an EgressNetwork (even set to Allow) and hostname labels: the proxy metrics show the route, the SNI or HTTP hostname, and the status.
  • How do you block egress with Linkerd? trafficPolicy: Deny, and allow destinations via TLS, HTTP or TCP routes with the EgressNetwork as parent and backend.
  • Is that a security boundary? No – Pods without a proxy bypass it. For hard boundaries, add network policies or an egress gateway.

Enterprise Note

  • Observe first, then block: run Allow with hostname labels for weeks, build the allowlist from that, then switch to Deny.
  • Hostname labels increase cardinality – enable them only where you need them, or aggregate them with a recording rule.
  • A global EgressNetwork in the linkerd-egress namespace sets a cluster-wide baseline rule; namespaces add their own allowances.
  • SNI allowances only check the name in the handshake. Against domain fronting, only a network policy on destination IPs or a proxy that breaks open TLS helps.
  • Egress routes belong in the app’s repo: a new destination is a code change with review.