Skip to content
Hands-on IT IT Drills

Day 6: Authorization

Goal: In notes, only those who must talk may talk: last → web → notes-api → postgres and web → helper. Everything else is denied, probes and Prometheus keep running. · Duration: approx. 75 min · Prerequisite: Day 5

Briefing

  • A Server describes a port of a group of Pods. Its accessPolicy applies to everything that matches no rule – default deny.
  • An AuthorizationPolicy connects a target (Server or route) with an authentication: MeshTLSAuthentication (identities) or NetworkAuthentication (IP networks, for plaintext).
  • Ports without a Server follow the proxy’s default policy (config.linkerd.io/default-inbound-policy), set when the Pod starts.
  • Order in operations: Server in audit mode, rules, evaluate, enforce.
  • Denials come back as HTTP 403; for TCP, the connection is closed.
 last ──> web ──> notes-api ──> postgres      allowed (MeshTLS)
           └────> helper                      allowed (MeshTLS)
 Kubelet ──> probes                           allowed automatically
 Prometheus (no proxy) ──> :4191              allowed (NetworkAuthentication)
 tag06/fremd, tag06/ohne ──> everything       403 or connection closed

Drill 1: Two intruders

fremd has a proxy and its own identity, ohne has none at all. Besides curl, both have a psql container.

mkdir -p ~/linkerd/tag06 && cd ~/linkerd/tag06
kubectl create namespace tag06
kubectl annotate namespace tag06 linkerd.io/inject=enabled
kubectl -n tag06 create serviceaccount fremd
cat > clients.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: fremd, namespace: tag06}
spec:
  replicas: 1
  selector: {matchLabels: {app: fremd}}
  template:
    metadata: {labels: {app: fremd}}
    spec:
      serviceAccountName: fremd
      containers:
      - name: curl
        image: docker.io/curlimages/curl:8.23.0
        command: [sleep, infinity]
      - name: psql
        image: docker.io/library/postgres:18.7-alpine
        command: [sleep, infinity]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: ohne, namespace: tag06}
spec:
  replicas: 1
  selector: {matchLabels: {app: ohne}}
  template:
    metadata:
      labels: {app: ohne}
      annotations: {linkerd.io/inject: disabled}
    spec:
      containers:
      - name: curl
        image: docker.io/curlimages/curl:8.23.0
        command: [sleep, infinity]
      - name: psql
        image: docker.io/library/postgres:18.7-alpine
        command: [sleep, infinity]
EOF
kubectl apply -f clients.yaml
kubectl -n tag06 rollout status deploy/fremd --timeout=120s
kubectl -n tag06 get pods

Expected (abridged):

NAME                    READY   STATUS    RESTARTS   AGE
fremd-5f65fcf95-x792g   3/3     Running   0          7s
ohne-ccfc5cbd9-jnvjf    2/2     Running   0          6s

Now both access everything notes has to offer:

for c in fremd ohne; do
  for u in web.notes/ notes-api.notes/notes helper.notes/get; do
    kubectl -n tag06 exec deploy/$c -c curl -- curl -s -o /dev/null \
      -w "$c $u %{http_code}\n" -H 'X-API-Key: lab-key-123' http://$u
  done
done
DB=postgres://notes:[email protected]:5432/notes
for c in fremd ohne; do
  kubectl -n tag06 exec deploy/$c -c psql -- psql $DB -tAc 'select count(*) from notes'
done

Expected:

fremd web.notes/ 200
fremd notes-api.notes/notes 200
fremd helper.notes/get 200
ohne web.notes/ 200
ohne notes-api.notes/notes 200
ohne helper.notes/get 200
1
1

This is the state after Day 5: everything encrypted, nothing locked. The API key is no protection – it sits in the configuration of web and is therefore available to anyone who can read it.

Save your progress? Sign up for free

Drill 2: Server in audit mode

linkerd viz authz -n notes deploy/helper

Expected (abridged):

ROUTE    SERVER                       AUTHORIZATION                UNAUTHORIZED  SUCCESS
default  default:all-unauthenticated  default/all-unauthenticated        0.0rps   79.66%
probe    default:all-unauthenticated  default/probe                      0.0rps  100.00%

No Server, so the default policy all-unauthenticated applies. The probe line is the readiness probes. Now one Server per port, initially with accessPolicy: audit – it blocks nothing, but flags everything that matches no rule. Server currently lives in policy.linkerd.io/v1beta3; older guides still show v1beta1.

cat > servers.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: web-http, namespace: notes}
spec:
  podSelector: {matchLabels: {app: web}}
  port: http
  proxyProtocol: HTTP/1
  accessPolicy: audit
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: notes-api-http, namespace: notes}
spec:
  podSelector: {matchLabels: {app: notes-api}}
  port: http
  proxyProtocol: HTTP/1
  accessPolicy: audit
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: helper-http, namespace: notes}
spec:
  podSelector: {matchLabels: {app: helper}}
  port: http
  proxyProtocol: HTTP/1
  accessPolicy: audit
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: postgres, namespace: notes}
spec:
  podSelector: {matchLabels: {app: postgres}}
  port: postgres
  proxyProtocol: opaque
  accessPolicy: audit
EOF
kubectl apply -f servers.yaml
kubectl -n notes get server
for c in fremd ohne; do
  kubectl -n tag06 exec deploy/$c -c curl -- \
    curl -s -o /dev/null http://helper.notes/get
done
sleep 70
linkerd viz authz -n notes deploy/helper
kubectl -n notes logs -l app=helper -c linkerd-proxy --since=2m \
  | grep 'Request allowed' \
  | grep -oE 'Name\("[^.]*\.[^.]*|client.tls=None' | sort | uniq -c

Expected (abridged):

NAME             PORT       PROTOCOL   ACCESS POLICY
helper-http      http       HTTP/1     audit
notes-api-http   http       HTTP/1     audit
postgres         postgres   opaque     audit
web-http         http       HTTP/1     audit
ROUTE    SERVER                       AUTHORIZATION                UNAUTHORIZED  SUCCESS
default  default:all-unauthenticated  default/all-unauthenticated        0.0rps  100.00%
probe    default:all-unauthenticated  default/probe                      0.0rps  100.00%
default  helper-http                  default/audit                      0.0rps   66.67%
probe    helper-http                  default/probe                      0.0rps  100.00%
      1 Name("fremd.tag06
     89 Name("web.notes
      1 client.tls=None

Port 8080 now belongs to the Server helper-http. The lines with default:all-unauthenticated are the admin port 4191: Prometheus scrapes and the proxy’s own probes. You take care of it in Drill 6. The log lists every request that only audit lets through: web (no rule yet), fremd with an identity, and ohne as client.tls=None – no proxy, no certificate.

Save your progress? Sign up for free

Drill 3: Allow identities

One MeshTLSAuthentication per client, one AuthorizationPolicy per edge. identityRefs to ServiceAccounts spares you the long identity names:

cat > authz.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: last, namespace: notes}
spec:
  identityRefs: [{kind: ServiceAccount, name: last}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: web, namespace: notes}
spec:
  identityRefs: [{kind: ServiceAccount, name: web}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: notes-api, namespace: notes}
spec:
  identityRefs: [{kind: ServiceAccount, name: notes-api}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: web-von-last, namespace: notes}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: web-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: last}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-von-web, namespace: notes}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: notes-api-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-von-web, namespace: notes}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: postgres-von-notes-api, namespace: notes}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: postgres}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: notes-api}
EOF
kubectl apply -f authz.yaml
sleep 65
linkerd viz authz -n notes deploy/helper | grep helper-http
linkerd viz authz -n notes deploy/web | grep web-http

Expected (abridged, numbers vary):

default  helper-http  authorizationpolicy/helper-von-web        0.0rps   68.18%  0.7rps
probe    helper-http  default/probe                             0.0rps  100.00%  0.2rps
default  web-http     authorizationpolicy/web-von-last          0.0rps   86.41%  1.7rps
default  web-http     default/audit                             0.0rps   91.67%  0.2rps

helper now sees web under its own rule. For web, however, part of the traffic still runs under audit – even though nobody except last is calling. Why?

for p in $(kubectl -n notes get pod -l app=web -o name); do
  linkerd diagnostics policy -n notes $p 8080 -o json \
    | jq -c '[.authorizations[].labels.name]'
done

Expected (the order may differ for you):

["audit","web-von-last"]
["web-von-last","audit"]

In audit mode, the Control Plane delivers audit as an additional authorization, and the order is not fixed. If audit comes first, the proxy also counts allowed traffic as audit. Remember: in audit mode, authz_name="audit" does not yet prove a violation – what matters is who shows up there.

Save your progress? Sign up for free

Drill 4: Evaluate the audit

First, let the intruders from Drill 1 knock once more – in audit mode they get through, but they are counted:

for c in fremd ohne; do
  for u in web.notes/ notes-api.notes/notes helper.notes/get; do
    kubectl -n tag06 exec deploy/$c -c curl -- curl -s -o /dev/null \
      -w "$c $u %{http_code}\n" -H 'X-API-Key: lab-key-123' http://$u
  done
done
sleep 30

Expected: six lines with 200, as in Drill 1.

Then query by client identity. Prometheus has been running since Day 4:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
Q='sum by (deployment, client_id) (increase(inbound_http_authz_allow_total{
 namespace="notes", authz_name="audit"}[10m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
  | jq -r '.data.result[] | .metric.deployment + " " + (.metric.client_id // "-")
  + " " + .value[1]'
kill %1

Expected (abridged, numbers vary):

helper web.notes.serviceaccount.identity.linkerd.cluster.local 212.4171857142857
helper fremd.tag06.serviceaccount.identity.linkerd.cluster.local 2.0825214285714284
helper - 2.0825214285714284
web last.notes.serviceaccount.identity.linkerd.cluster.local 639.4215133126947
notes-api web.notes.serviceaccount.identity.linkerd.cluster.local 423.33642857142854
web fremd.tag06.serviceaccount.identity.linkerd.cluster.local 0
notes-api fremd.tag06.serviceaccount.identity.linkerd.cluster.local 0
web - 0
notes-api - 0

The lines with web and last from notes match the rules (Drill 3). fremd and - (no identity, that is, ohne) are your intruders – exactly these should fail in a moment. The 0 does not mean “no access”: the series is created with the first request, and increase() only counts what comes after it. At helper, both already knocked in Drill 2. If there were a client here that you do not know, now would be the moment to allow it or to find out what it is.

Postgres is opaque, so TCP. For that, the proxy counts connections:

linkerd diagnostics proxy-metrics -n notes po/postgres-0 \
  | grep '^inbound_tcp_authz_allow_total' \
  | grep -oE '(authz_name|client_id)="[^"]*"|\} [0-9]+' | paste - - -

Expected (abridged):

authz_name="postgres-von-notes-api"  client_id="notes-api.notes.serviceacco...  } 7

Save your progress? Sign up for free

Drill 5: Enforce

sed -i 's/accessPolicy: audit/accessPolicy: deny/' servers.yaml
kubectl apply -f servers.yaml
kubectl -n notes get server
for c in fremd ohne; do
  for u in web.notes/ notes-api.notes/notes helper.notes/get; do
    kubectl -n tag06 exec deploy/$c -c curl -- curl -s -o /dev/null \
      -w "$c $u %{http_code}\n" -H 'X-API-Key: lab-key-123' http://$u
  done
done
for c in fremd ohne; do
  kubectl -n tag06 exec deploy/$c -c psql -- psql $DB -tAc 'select count(*) from notes'
done

Expected (abridged):

NAME             PORT       PROTOCOL   ACCESS POLICY
helper-http      http       HTTP/1     deny
...
fremd web.notes/ 403
fremd notes-api.notes/notes 403
fremd helper.notes/get 403
ohne web.notes/ 403
ohne notes-api.notes/notes 403
ohne helper.notes/get 403
psql: error: connection to server at "postgres.notes" (10.109.208.153), port 5432
failed: server closed the connection unexpectedly
psql: error: connection to server at "postgres.notes" (10.109.208.153), port 5432
failed: server closed the connection unexpectedly

HTTP is answered with 403 – for ohne too, because the proxy of web recognizes HTTP and answers itself. Postgres is opaque: there, the proxy closes the connection. And the legitimate traffic?

sleep 50
kubectl -n notes logs deploy/last -c last --since=45s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (numbers vary):

     64 /api/notes 200
     22 /flaky 200
     10 /flaky 500

Unchanged. Now make the denials visible:

kubectl -n tag06 exec deploy/fremd -c curl -- sh -c 'for i in $(seq 60); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper.notes/get; sleep 0.5
  done' | sort | uniq -c
sleep 20
linkerd viz authz -n notes deploy/helper | grep helper-http
linkerd diagnostics proxy-metrics -n notes po/postgres-0 \
  | grep '^inbound_tcp_authz_deny_total' \
  | grep -oE '(client_id|tls|no_tls_reason)="[^"]*"'

Expected (abridged):

     60 403
default  helper-http                                                  1.1rps    0.00%
default  helper-http  authorizationpolicy/helper-von-web              0.0rps   69.05%
probe    helper-http  default/probe                                   0.0rps  100.00%
tls="no_identity"
no_tls_reason="port_skipped"
tls="true"
client_id="fremd.tag06.serviceaccount.identity.linkerd.cluster.local"

The UNAUTHORIZED column shows the denied requests from fremd. At Postgres, fremd appears with its identity, ohne as plaintext on the opaque port (port_skipped, as on Day 3). The probes keep running: as long as a Server has no routes, Linkerd automatically allows the probe paths from the Pod spec. The proxy log records every denial with the identity:

kubectl -n notes logs deploy/helper -c linkerd-proxy --since=2m \
  | grep 'Request denied' | tail -1 | grep -oE 'server.name=[^ ]*|Name\("[^"]*'

Expected:

server.name=helper-http
Name("fremd.tag06.serviceaccount.identity.linkerd.cluster.local

Save your progress? Sign up for free

Drill 6: Default deny for the rest

The Servers only cover the app ports. Every other port, including a new one, follows the default policy. You set it to deny for the whole namespace. It only takes effect for new Pods:

kubectl annotate namespace notes config.linkerd.io/default-inbound-policy=deny
kubectl -n notes rollout restart deploy/web deploy/helper deploy/notes-api deploy/last
kubectl -n notes rollout restart statefulset/postgres
kubectl -n notes rollout status deploy/notes-api --timeout=180s
kubectl -n notes rollout status statefulset/postgres --timeout=180s
kubectl -n notes get pod -l app=web -o yaml | grep -m1 -A1 INBOUND_DEFAULT_POLICY

Expected:

      - name: LINKERD2_PROXY_INBOUND_DEFAULT_POLICY
        value: deny

All Pods become 2/2 – the proxy’s probes on 4191 are also allowed automatically. But after a minute:

sleep 60
linkerd viz authz -n notes deploy/helper
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
curl -s localhost:9090/api/v1/targets?state=active | jq -r '.data.activeTargets[]
  | select(.labels.namespace=="notes") | .labels.pod + " " + .lastError' | head -3

Expected (abridged):

ROUTE  SERVER  AUTHORIZATION  UNAUTHORIZED  SUCCESS  RPS  LATENCY_P50  LATENCY_P95
helper-55bb5dfbbb-xqb59 server returned HTTP status 403 Forbidden
web-5967795fd8-8kzww server returned HTTP status 403 Forbidden
postgres-0 server returned HTTP status 403 Forbidden

Prometheus runs without a proxy and scrapes 4191 in plaintext – forbidden under deny. viz and Grafana are empty. That is what NetworkAuthentication is for. The Pod IP of Prometheus changes, so you allow the Pod network (with Cilium in the lab 10.0.0.0/16, for you: kubectl get ciliumnodes -o jsonpath='{..podCIDRs}'):

cat > admin.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: proxy-admin, namespace: notes}
spec:
  podSelector: {}
  port: linkerd-admin
  proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: pod-netz, namespace: notes}
spec:
  networks:
  - cidr: 10.0.0.0/16
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: admin-von-prometheus, namespace: notes}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: proxy-admin}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: NetworkAuthentication, name: pod-netz}
EOF
kubectl apply -f admin.yaml
sleep 30
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=up{job="monitoring/linkerd-proxy", namespace="notes"}' \
  | jq -r '.data.result[] | .metric.pod + " " + .value[1]' | head -3
kill %1

Expected (abridged):

web-5967795fd8-gr4fz 1
postgres-0 1
helper-55bb5dfbbb-xqb59 1

The permission only applies to Pods in notes – Server and AuthorizationPolicy never reach beyond the namespace. If a Pod in another namespace has its own Server for linkerd-admin (such as the multicluster gateway on Day 12), it rejects the scrape with 403 until a separate rule for Prometheus exists there.

Warning: In production, you create proxy-admin before you switch the default policy – otherwise you lose minutes of metrics. And 10.0.0.0/16 means: every Pod in the cluster may read /metrics. Tighter is only possible with Prometheus in the mesh and a MeshTLSAuthentication on its identity.

linkerd viz tap keeps working without its own rule: the tap port 4190 checks by itself that the caller has the identity of tap from linkerd-viz. linkerd identity and linkerd diagnostics keep working as well – they come in via port-forward over localhost, which the proxy always allows.

Save your progress? Sign up for free

Drill 7: The init container from Day 2

On Day 2, migrate ran before the proxy without a native sidecar. Now with the identity of notes-api, that is, with all permissions – in theory:

cat > migrate-alt.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: migrate-alt, namespace: notes}
spec:
  replicas: 1
  selector: {matchLabels: {app: migrate-alt}}
  template:
    metadata:
      labels: {app: migrate-alt}
      annotations:
        config.beta.linkerd.io/proxy-enable-native-sidecar: "false"
    spec:
      serviceAccountName: notes-api
      initContainers:
      - name: migrate
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        args: [migrate]
        env:
        - name: NOTES_DB_URL
          valueFrom: {secretKeyRef: {name: notes-db, key: url}}
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.10
EOF
kubectl apply -f migrate-alt.yaml
sleep 40
kubectl -n notes get pod -l app=migrate-alt
kubectl -n notes logs deploy/migrate-alt -c migrate --tail=1
linkerd diagnostics proxy-metrics -n notes po/postgres-0 \
  | grep '^inbound_tcp_authz_deny_total' | grep -oE 'no_tls_reason="[^"]*"|\} [0-9]+'

Expected (abridged):

NAME                           READY   STATUS       RESTARTS      AGE
migrate-alt-5c97994d8b-s24vd   0/2     Init:Error   2 (34s ago)   40s
time=2026-11-19T16:22:50.801Z level=ERROR msg="database unreachable" source=env
err="failed to connect to `user=notes database=notes`: 10.109.208.153:5432 (postgres):
failed to receive message: unexpected EOF"
no_tls_reason="port_skipped"
} 5

The right ServiceAccount does not help: migrate runs before there is a proxy, so without a certificate. Postgres sees plaintext and denies it. With a native sidecar:

kubectl -n notes patch deploy migrate-alt --type=json \
  -p '[{"op":"remove","path":"/spec/template/metadata/annotations"}]'
kubectl -n notes rollout status deploy/migrate-alt --timeout=120s
kubectl -n notes logs deploy/migrate-alt -c migrate --tail=1
kubectl -n notes delete deploy migrate-alt

Expected (abridged):

deployment "migrate-alt" successfully rolled out
time=2026-11-19T16:23:26.437Z level=INFO msg="migration complete" table=notes

Save your progress? Sign up for free

Drill 8: Taking stock

kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n notes deploy
linkerd viz authz -n notes deploy/web
linkerd check --proxy -n notes 2>&1 | tail -1

Expected (abridged, numbers vary):

     80 /api/notes 200
     29 /flaky 200
     11 /flaky 500
NAME        MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
helper         2/2    86.32%   1.6rps           1ms          32ms          38ms
last           1/1   100.00%   0.3rps           1ms          49ms          50ms
notes-api      2/2   100.00%   2.2rps           6ms          39ms          89ms
web            2/2    92.12%   2.8rps          14ms          92ms         170ms
ROUTE    SERVER       AUTHORIZATION                             UNAUTHORIZED  SUCCESS
default  proxy-admin  authorizationpolicy/admin-von-prometheus        0.0rps  100.00%
probe    proxy-admin  default/probe                                   0.0rps  100.00%
default  web-http     authorizationpolicy/web-von-last                0.0rps   89.92%
Status check results are √

Same rates as on Day 0, no more audit, no more default:all-unauthenticated. In the Grafana dashboard “Linkerd Deployment” you see a gap of a few minutes (Drill 6), after which the curves continue. These policies stay active for the rest of the training.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all policy resources in notes with one command. Target time: 20 s
  2. Show which default policy the proxy of helper has. Target time: 45 s
  3. Show which authorizations the proxy of notes-api knows for port 8080. Target time: 45 s
  4. Prove that fremd can no longer get into the database. Target time: 30 s

Solution

kubectl -n notes get server,authorizationpolicy,meshtlsauthentication,\
networkauthentication
kubectl -n notes get pod -l app=helper -o yaml | grep -m1 -A1 INBOUND_DEFAULT_POLICY
P=$(kubectl -n notes get pod -l app=notes-api -o name | head -1)
linkerd diagnostics policy -n notes $P 8080 -o json \
  | jq -r '.authorizations[].labels.name'
kubectl -n tag06 exec deploy/fremd -c psql -- psql $DB -tAc 'select 1'

Expected (abridged):

NAME                                      PORT            PROTOCOL   ACCESS POLICY
server.policy.linkerd.io/helper-http      http            HTTP/1     deny
server.policy.linkerd.io/notes-api-http   http            HTTP/1     deny
server.policy.linkerd.io/postgres         postgres        opaque     deny
server.policy.linkerd.io/proxy-admin      linkerd-admin   HTTP/1     deny
server.policy.linkerd.io/web-http         http            HTTP/1     deny
      - name: LINKERD2_PROXY_INBOUND_DEFAULT_POLICY
        value: deny
notes-api-von-web
psql: error: connection to server at "postgres.notes" (10.109.208.153), port 5432
failed: server closed the connection unexpectedly

linkerd diagnostics policy wants a Pod or a Service, not a Deployment.

Save your progress? Sign up for free

Checkpoint

kubectl -n tag06 exec deploy/fremd -c curl -- curl -s -o /dev/null \
  -w '%{http_code}\n' http://helper.notes/get

Expected: 403

kubectl -n notes get server -o custom-columns=NAME:.metadata.name,\
POLICY:.spec.accessPolicy

Expected: five Servers, all deny (with no value set, proxy-admin defaults to deny).

kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected: /api/notes only 200, /flaky about 70 percent 200.

Challenge

In tag06, create a copy of helper (ServiceAccount helper, one replica) and allow fremd only GET /get there. The readiness probe must not break.

Hint: As soon as a route is attached to a Server, Linkerd no longer allows the probes by itself – and the kubelet has no identity.

Solution

kubectl -n tag06 create serviceaccount helper
sed -e 's/namespace: notes/namespace: tag06/' -e 's/replicas: 2/replicas: 1/' \
  ~/linkerd/tag00/helper.yaml > helper-kopie.yaml
kubectl apply -f helper-kopie.yaml
kubectl -n tag06 set serviceaccount deploy/helper helper
kubectl -n tag06 rollout status deploy/helper --timeout=120s
cat > challenge.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: helper-http, namespace: tag06}
spec:
  podSelector: {matchLabels: {app: helper}}
  port: http
  proxyProtocol: HTTP/1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: helper-get, namespace: tag06}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: Server, name: helper-http}
  rules:
  - matches:
    - path: {type: Exact, value: /get}
      method: GET
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: fremd, namespace: tag06}
spec:
  identityRefs: [{kind: ServiceAccount, name: fremd}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-get-von-fremd, namespace: tag06}
spec:
  targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: helper-get}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: fremd}
EOF
kubectl apply -f challenge.yaml
sleep 45
kubectl -n tag06 get pods -l app=helper
kubectl -n tag06 get events --field-selector reason=Unhealthy | tail -1

Expected (abridged):

NAME                      READY   STATUS    RESTARTS   AGE
helper-79d57fbbbb-xx9jg   1/2     Running   0          54s
8s   Warning   Unhealthy   pod/helper-79d57fbbbb-xx9jg   Readiness probe failed: HTTP
probe failed with statuscode: 404

The probe /status/200 no longer matches any route: 404. So you need a route for the probe and a rule for the kubelet. Where it comes from is shown in the log as soon as the route exists:

cat > probe-route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: helper-probe, namespace: tag06}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: Server, name: helper-http}
  rules:
  - matches:
    - path: {type: Exact, value: /status/200}
      method: GET
EOF
kubectl apply -f probe-route.yaml
sleep 25
kubectl -n tag06 logs deploy/helper -c linkerd-proxy --since=20s \
  | grep 'Request denied' | grep -oE 'route.name=[^ ]*|client.ip=[0-9.]*' | sort -u
kubectl get ciliumnodes \
  -o jsonpath='{..addresses[?(@.type=="CiliumInternalIP")].ip}{"\n"}'

Expected:

client.ip=10.0.1.116
route.name=helper-probe
10.0.0.88 10.0.1.116 10.0.2.241

With Cilium, the kubelet comes in via the node’s cilium_host address. You enter those (your IPs will differ):

cat > kubelet.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: kubelet, namespace: tag06}
spec:
  networks:
  - cidr: 10.0.0.88/32
  - cidr: 10.0.1.116/32
  - cidr: 10.0.2.241/32
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-probe-von-kubelet, namespace: tag06}
spec:
  targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: helper-probe}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: NetworkAuthentication, name: kubelet}
EOF
kubectl apply -f kubelet.yaml
sleep 25
kubectl -n tag06 get pods -l app=helper
for u in /get /status/200 /headers; do
  kubectl -n tag06 exec deploy/fremd -c curl -- curl -s -o /dev/null \
    -w "$u %{http_code}\n" http://helper.tag06$u
done

Expected:

NAME                      READY   STATUS    RESTARTS   AGE
helper-79d57fbbbb-xx9jg   2/2     Running   0          2m3s
/get 200
/status/200 403
/headers 404

/get is allowed, the probe only for the kubelet, everything else matches no route (404).

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag06

The policies in notes stay.

Interview Note

  • How do you introduce authorization without an outage? Servers with accessPolicy: audit, create rules, evaluate the audit by client identity, then deny. Finally, set the namespace’s default policy to deny for new ports.
  • What happens to probes and Prometheus? Linkerd allows probes automatically as long as the Server has no routes. Prometheus without a proxy needs a NetworkAuthentication for the admin port 4191.
  • 403 or a dropped connection? HTTP gets a 403 from the proxy, opaque/TCP is closed.

Enterprise Note

  • Policies belong in Git next to the Deployments; every new edge is a review.
  • Set the default policy cluster-wide via Helm (proxy.defaultInboundPolicy) instead of per namespace – but it only takes effect after all Pods restart.
  • Bring Prometheus into the mesh and allow it by identity instead of opening up the Pod network.
  • Alert on inbound_http_authz_deny_total and inbound_tcp_authz_deny_total: a rise is either an attack or a forgotten legitimate client.
  • Run jobs, migrations and backups with a native sidecar; otherwise they hit a wall under default deny.