Day 6: Authorization
Goal: In
notes, only those who must talk may talk:last → web → notes-api → postgresandweb → helper. Everything else is denied, probes and Prometheus keep running. · Duration: approx. 75 min · Prerequisite: Day 5
Briefing
- A Server describes a port of a group of Pods. Its
accessPolicyapplies to everything that matches no rule – defaultdeny. - An AuthorizationPolicy connects a target (Server or route) with an authentication: MeshTLSAuthentication (identities) or NetworkAuthentication (IP networks, for plaintext).
- Ports without a Server follow the proxy’s default
policy (
config.linkerd.io/default-inbound-policy), set when the Pod starts. - Order in operations: Server in audit mode, rules, evaluate, enforce.
- Denials come back as HTTP 403; for TCP, the connection is closed.
last ──> web ──> notes-api ──> postgres allowed (MeshTLS)
└────> helper allowed (MeshTLS)
Kubelet ──> probes allowed automatically
Prometheus (no proxy) ──> :4191 allowed (NetworkAuthentication)
tag06/fremd, tag06/ohne ──> everything 403 or connection closed
Drill 1: Two intruders
fremd has a proxy and its own identity,
ohne has none at all. Besides curl, both have
a psql container.
mkdir -p ~/linkerd/tag06 && cd ~/linkerd/tag06
kubectl create namespace tag06
kubectl annotate namespace tag06 linkerd.io/inject=enabled
kubectl -n tag06 create serviceaccount fremd
cat > clients.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: fremd, namespace: tag06}
spec:
replicas: 1
selector: {matchLabels: {app: fremd}}
template:
metadata: {labels: {app: fremd}}
spec:
serviceAccountName: fremd
containers:
- name: curl
image: docker.io/curlimages/curl:8.23.0
command: [sleep, infinity]
- name: psql
image: docker.io/library/postgres:18.7-alpine
command: [sleep, infinity]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: ohne, namespace: tag06}
spec:
replicas: 1
selector: {matchLabels: {app: ohne}}
template:
metadata:
labels: {app: ohne}
annotations: {linkerd.io/inject: disabled}
spec:
containers:
- name: curl
image: docker.io/curlimages/curl:8.23.0
command: [sleep, infinity]
- name: psql
image: docker.io/library/postgres:18.7-alpine
command: [sleep, infinity]
EOF
kubectl apply -f clients.yaml
kubectl -n tag06 rollout status deploy/fremd --timeout=120s
kubectl -n tag06 get podsExpected (abridged):
NAME READY STATUS RESTARTS AGE
fremd-5f65fcf95-x792g 3/3 Running 0 7s
ohne-ccfc5cbd9-jnvjf 2/2 Running 0 6s
Now both access everything notes has to offer:
for c in fremd ohne; do
for u in web.notes/ notes-api.notes/notes helper.notes/get; do
kubectl -n tag06 exec deploy/$c -c curl -- curl -s -o /dev/null \
-w "$c $u %{http_code}\n" -H 'X-API-Key: lab-key-123' http://$u
done
done
DB=postgres://notes:[email protected]:5432/notes
for c in fremd ohne; do
kubectl -n tag06 exec deploy/$c -c psql -- psql $DB -tAc 'select count(*) from notes'
doneExpected:
fremd web.notes/ 200
fremd notes-api.notes/notes 200
fremd helper.notes/get 200
ohne web.notes/ 200
ohne notes-api.notes/notes 200
ohne helper.notes/get 200
1
1
This is the state after Day 5: everything encrypted, nothing locked.
The API key is no protection – it sits in the configuration of
web and is therefore available to anyone who can read
it.
Save your progress? Sign up for free
Drill 2: Server in audit mode
linkerd viz authz -n notes deploy/helperExpected (abridged):
ROUTE SERVER AUTHORIZATION UNAUTHORIZED SUCCESS
default default:all-unauthenticated default/all-unauthenticated 0.0rps 79.66%
probe default:all-unauthenticated default/probe 0.0rps 100.00%
No Server, so the default policy all-unauthenticated
applies. The probe line is the readiness probes. Now one
Server per port, initially with accessPolicy: audit – it
blocks nothing, but flags everything that matches no rule.
Server currently lives in
policy.linkerd.io/v1beta3; older guides still show
v1beta1.
cat > servers.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: web-http, namespace: notes}
spec:
podSelector: {matchLabels: {app: web}}
port: http
proxyProtocol: HTTP/1
accessPolicy: audit
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: notes-api-http, namespace: notes}
spec:
podSelector: {matchLabels: {app: notes-api}}
port: http
proxyProtocol: HTTP/1
accessPolicy: audit
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: helper-http, namespace: notes}
spec:
podSelector: {matchLabels: {app: helper}}
port: http
proxyProtocol: HTTP/1
accessPolicy: audit
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: postgres, namespace: notes}
spec:
podSelector: {matchLabels: {app: postgres}}
port: postgres
proxyProtocol: opaque
accessPolicy: audit
EOF
kubectl apply -f servers.yaml
kubectl -n notes get server
for c in fremd ohne; do
kubectl -n tag06 exec deploy/$c -c curl -- \
curl -s -o /dev/null http://helper.notes/get
done
sleep 70
linkerd viz authz -n notes deploy/helper
kubectl -n notes logs -l app=helper -c linkerd-proxy --since=2m \
| grep 'Request allowed' \
| grep -oE 'Name\("[^.]*\.[^.]*|client.tls=None' | sort | uniq -cExpected (abridged):
NAME PORT PROTOCOL ACCESS POLICY
helper-http http HTTP/1 audit
notes-api-http http HTTP/1 audit
postgres postgres opaque audit
web-http http HTTP/1 audit
ROUTE SERVER AUTHORIZATION UNAUTHORIZED SUCCESS
default default:all-unauthenticated default/all-unauthenticated 0.0rps 100.00%
probe default:all-unauthenticated default/probe 0.0rps 100.00%
default helper-http default/audit 0.0rps 66.67%
probe helper-http default/probe 0.0rps 100.00%
1 Name("fremd.tag06
89 Name("web.notes
1 client.tls=None
Port 8080 now belongs to the Server helper-http. The
lines with default:all-unauthenticated are the admin port
4191: Prometheus scrapes and the proxy’s own probes. You take care of it
in Drill 6. The log lists every request that only audit
lets through: web (no rule yet), fremd with an
identity, and ohne as client.tls=None – no
proxy, no certificate.
Save your progress? Sign up for free
Drill 3: Allow identities
One MeshTLSAuthentication per client, one AuthorizationPolicy per
edge. identityRefs to ServiceAccounts spares you the long
identity names:
cat > authz.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: last, namespace: notes}
spec:
identityRefs: [{kind: ServiceAccount, name: last}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: web, namespace: notes}
spec:
identityRefs: [{kind: ServiceAccount, name: web}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: notes-api, namespace: notes}
spec:
identityRefs: [{kind: ServiceAccount, name: notes-api}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: web-von-last, namespace: notes}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: web-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: last}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-von-web, namespace: notes}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: notes-api-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-von-web, namespace: notes}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: postgres-von-notes-api, namespace: notes}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: postgres}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: notes-api}
EOF
kubectl apply -f authz.yaml
sleep 65
linkerd viz authz -n notes deploy/helper | grep helper-http
linkerd viz authz -n notes deploy/web | grep web-httpExpected (abridged, numbers vary):
default helper-http authorizationpolicy/helper-von-web 0.0rps 68.18% 0.7rps
probe helper-http default/probe 0.0rps 100.00% 0.2rps
default web-http authorizationpolicy/web-von-last 0.0rps 86.41% 1.7rps
default web-http default/audit 0.0rps 91.67% 0.2rps
helper now sees web under its own rule. For
web, however, part of the traffic still runs under
audit – even though nobody except last is
calling. Why?
for p in $(kubectl -n notes get pod -l app=web -o name); do
linkerd diagnostics policy -n notes $p 8080 -o json \
| jq -c '[.authorizations[].labels.name]'
doneExpected (the order may differ for you):
["audit","web-von-last"]
["web-von-last","audit"]
In audit mode, the Control Plane delivers audit as an
additional authorization, and the order is not fixed. If
audit comes first, the proxy also counts allowed traffic as
audit. Remember: in audit mode,
authz_name="audit" does not yet prove a violation – what
matters is who shows up there.
Save your progress? Sign up for free
Drill 4: Evaluate the audit
First, let the intruders from Drill 1 knock once more – in audit mode they get through, but they are counted:
for c in fremd ohne; do
for u in web.notes/ notes-api.notes/notes helper.notes/get; do
kubectl -n tag06 exec deploy/$c -c curl -- curl -s -o /dev/null \
-w "$c $u %{http_code}\n" -H 'X-API-Key: lab-key-123' http://$u
done
done
sleep 30Expected: six lines with 200, as in
Drill 1.
Then query by client identity. Prometheus has been running since Day 4:
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
Q='sum by (deployment, client_id) (increase(inbound_http_authz_allow_total{
namespace="notes", authz_name="audit"}[10m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
| jq -r '.data.result[] | .metric.deployment + " " + (.metric.client_id // "-")
+ " " + .value[1]'
kill %1Expected (abridged, numbers vary):
helper web.notes.serviceaccount.identity.linkerd.cluster.local 212.4171857142857
helper fremd.tag06.serviceaccount.identity.linkerd.cluster.local 2.0825214285714284
helper - 2.0825214285714284
web last.notes.serviceaccount.identity.linkerd.cluster.local 639.4215133126947
notes-api web.notes.serviceaccount.identity.linkerd.cluster.local 423.33642857142854
web fremd.tag06.serviceaccount.identity.linkerd.cluster.local 0
notes-api fremd.tag06.serviceaccount.identity.linkerd.cluster.local 0
web - 0
notes-api - 0
The lines with web and last from
notes match the rules (Drill 3). fremd and
- (no identity, that is, ohne) are your
intruders – exactly these should fail in a moment. The 0
does not mean “no access”: the series is created with the first request,
and increase() only counts what comes after it. At
helper, both already knocked in Drill 2. If there were a
client here that you do not know, now would be the moment to allow it or
to find out what it is.
Postgres is opaque, so TCP. For that, the proxy counts connections:
linkerd diagnostics proxy-metrics -n notes po/postgres-0 \
| grep '^inbound_tcp_authz_allow_total' \
| grep -oE '(authz_name|client_id)="[^"]*"|\} [0-9]+' | paste - - -Expected (abridged):
authz_name="postgres-von-notes-api" client_id="notes-api.notes.serviceacco... } 7
Save your progress? Sign up for free
Drill 5: Enforce
sed -i 's/accessPolicy: audit/accessPolicy: deny/' servers.yaml
kubectl apply -f servers.yaml
kubectl -n notes get server
for c in fremd ohne; do
for u in web.notes/ notes-api.notes/notes helper.notes/get; do
kubectl -n tag06 exec deploy/$c -c curl -- curl -s -o /dev/null \
-w "$c $u %{http_code}\n" -H 'X-API-Key: lab-key-123' http://$u
done
done
for c in fremd ohne; do
kubectl -n tag06 exec deploy/$c -c psql -- psql $DB -tAc 'select count(*) from notes'
doneExpected (abridged):
NAME PORT PROTOCOL ACCESS POLICY
helper-http http HTTP/1 deny
...
fremd web.notes/ 403
fremd notes-api.notes/notes 403
fremd helper.notes/get 403
ohne web.notes/ 403
ohne notes-api.notes/notes 403
ohne helper.notes/get 403
psql: error: connection to server at "postgres.notes" (10.109.208.153), port 5432
failed: server closed the connection unexpectedly
psql: error: connection to server at "postgres.notes" (10.109.208.153), port 5432
failed: server closed the connection unexpectedly
HTTP is answered with 403 – for ohne too, because the
proxy of web recognizes HTTP and answers itself. Postgres
is opaque: there, the proxy closes the connection. And the legitimate
traffic?
sleep 50
kubectl -n notes logs deploy/last -c last --since=45s | awk '{print $2, $3}' \
| sort | uniq -cExpected (numbers vary):
64 /api/notes 200
22 /flaky 200
10 /flaky 500
Unchanged. Now make the denials visible:
kubectl -n tag06 exec deploy/fremd -c curl -- sh -c 'for i in $(seq 60); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper.notes/get; sleep 0.5
done' | sort | uniq -c
sleep 20
linkerd viz authz -n notes deploy/helper | grep helper-http
linkerd diagnostics proxy-metrics -n notes po/postgres-0 \
| grep '^inbound_tcp_authz_deny_total' \
| grep -oE '(client_id|tls|no_tls_reason)="[^"]*"'Expected (abridged):
60 403
default helper-http 1.1rps 0.00%
default helper-http authorizationpolicy/helper-von-web 0.0rps 69.05%
probe helper-http default/probe 0.0rps 100.00%
tls="no_identity"
no_tls_reason="port_skipped"
tls="true"
client_id="fremd.tag06.serviceaccount.identity.linkerd.cluster.local"
The UNAUTHORIZED column shows the denied requests from
fremd. At Postgres, fremd appears with its
identity, ohne as plaintext on the opaque port
(port_skipped, as on Day 3). The probes keep running: as
long as a Server has no routes, Linkerd automatically allows the probe
paths from the Pod spec. The proxy log records every denial with the
identity:
kubectl -n notes logs deploy/helper -c linkerd-proxy --since=2m \
| grep 'Request denied' | tail -1 | grep -oE 'server.name=[^ ]*|Name\("[^"]*'Expected:
server.name=helper-http
Name("fremd.tag06.serviceaccount.identity.linkerd.cluster.local
Save your progress? Sign up for free
Drill 6: Default deny for the rest
The Servers only cover the app ports. Every other port, including a
new one, follows the default policy. You set it to deny for
the whole namespace. It only takes effect for new Pods:
kubectl annotate namespace notes config.linkerd.io/default-inbound-policy=deny
kubectl -n notes rollout restart deploy/web deploy/helper deploy/notes-api deploy/last
kubectl -n notes rollout restart statefulset/postgres
kubectl -n notes rollout status deploy/notes-api --timeout=180s
kubectl -n notes rollout status statefulset/postgres --timeout=180s
kubectl -n notes get pod -l app=web -o yaml | grep -m1 -A1 INBOUND_DEFAULT_POLICYExpected:
- name: LINKERD2_PROXY_INBOUND_DEFAULT_POLICY
value: deny
All Pods become 2/2 – the proxy’s probes on 4191 are
also allowed automatically. But after a minute:
sleep 60
linkerd viz authz -n notes deploy/helper
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
curl -s localhost:9090/api/v1/targets?state=active | jq -r '.data.activeTargets[]
| select(.labels.namespace=="notes") | .labels.pod + " " + .lastError' | head -3Expected (abridged):
ROUTE SERVER AUTHORIZATION UNAUTHORIZED SUCCESS RPS LATENCY_P50 LATENCY_P95
helper-55bb5dfbbb-xqb59 server returned HTTP status 403 Forbidden
web-5967795fd8-8kzww server returned HTTP status 403 Forbidden
postgres-0 server returned HTTP status 403 Forbidden
Prometheus runs without a proxy and scrapes 4191 in plaintext –
forbidden under deny. viz and Grafana are empty. That is
what NetworkAuthentication is for. The Pod IP of Prometheus changes, so
you allow the Pod network (with Cilium in the lab
10.0.0.0/16, for you:
kubectl get ciliumnodes -o jsonpath='{..podCIDRs}'):
cat > admin.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: proxy-admin, namespace: notes}
spec:
podSelector: {}
port: linkerd-admin
proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: pod-netz, namespace: notes}
spec:
networks:
- cidr: 10.0.0.0/16
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: admin-von-prometheus, namespace: notes}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: proxy-admin}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: NetworkAuthentication, name: pod-netz}
EOF
kubectl apply -f admin.yaml
sleep 30
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=up{job="monitoring/linkerd-proxy", namespace="notes"}' \
| jq -r '.data.result[] | .metric.pod + " " + .value[1]' | head -3
kill %1Expected (abridged):
web-5967795fd8-gr4fz 1
postgres-0 1
helper-55bb5dfbbb-xqb59 1
The permission only applies to Pods in notes – Server
and AuthorizationPolicy never reach beyond the namespace. If a Pod in
another namespace has its own Server for linkerd-admin
(such as the multicluster gateway on Day 12), it rejects the scrape with
403 until a separate rule for Prometheus exists there.
Warning: In production, you create
proxy-adminbefore you switch the default policy – otherwise you lose minutes of metrics. And10.0.0.0/16means: every Pod in the cluster may read/metrics. Tighter is only possible with Prometheus in the mesh and a MeshTLSAuthentication on its identity.
linkerd viz tap keeps working without its own rule: the
tap port 4190 checks by itself that the caller has the identity of
tap from linkerd-viz.
linkerd identity and linkerd diagnostics keep
working as well – they come in via port-forward over
localhost, which the proxy always allows.
Save your progress? Sign up for free
Drill 7: The init container from Day 2
On Day 2, migrate ran before the proxy without a native
sidecar. Now with the identity of notes-api, that is, with
all permissions – in theory:
cat > migrate-alt.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: migrate-alt, namespace: notes}
spec:
replicas: 1
selector: {matchLabels: {app: migrate-alt}}
template:
metadata:
labels: {app: migrate-alt}
annotations:
config.beta.linkerd.io/proxy-enable-native-sidecar: "false"
spec:
serviceAccountName: notes-api
initContainers:
- name: migrate
image: ghcr.io/thomaszachmann/notes-api:0.7.1
args: [migrate]
env:
- name: NOTES_DB_URL
valueFrom: {secretKeyRef: {name: notes-db, key: url}}
containers:
- name: pause
image: registry.k8s.io/pause:3.10
EOF
kubectl apply -f migrate-alt.yaml
sleep 40
kubectl -n notes get pod -l app=migrate-alt
kubectl -n notes logs deploy/migrate-alt -c migrate --tail=1
linkerd diagnostics proxy-metrics -n notes po/postgres-0 \
| grep '^inbound_tcp_authz_deny_total' | grep -oE 'no_tls_reason="[^"]*"|\} [0-9]+'Expected (abridged):
NAME READY STATUS RESTARTS AGE
migrate-alt-5c97994d8b-s24vd 0/2 Init:Error 2 (34s ago) 40s
time=2026-11-19T16:22:50.801Z level=ERROR msg="database unreachable" source=env
err="failed to connect to `user=notes database=notes`: 10.109.208.153:5432 (postgres):
failed to receive message: unexpected EOF"
no_tls_reason="port_skipped"
} 5
The right ServiceAccount does not help: migrate runs
before there is a proxy, so without a certificate. Postgres sees
plaintext and denies it. With a native sidecar:
kubectl -n notes patch deploy migrate-alt --type=json \
-p '[{"op":"remove","path":"/spec/template/metadata/annotations"}]'
kubectl -n notes rollout status deploy/migrate-alt --timeout=120s
kubectl -n notes logs deploy/migrate-alt -c migrate --tail=1
kubectl -n notes delete deploy migrate-altExpected (abridged):
deployment "migrate-alt" successfully rolled out
time=2026-11-19T16:23:26.437Z level=INFO msg="migration complete" table=notes
Save your progress? Sign up for free
Drill 8: Taking stock
kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n notes deploy
linkerd viz authz -n notes deploy/web
linkerd check --proxy -n notes 2>&1 | tail -1Expected (abridged, numbers vary):
80 /api/notes 200
29 /flaky 200
11 /flaky 500
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99
helper 2/2 86.32% 1.6rps 1ms 32ms 38ms
last 1/1 100.00% 0.3rps 1ms 49ms 50ms
notes-api 2/2 100.00% 2.2rps 6ms 39ms 89ms
web 2/2 92.12% 2.8rps 14ms 92ms 170ms
ROUTE SERVER AUTHORIZATION UNAUTHORIZED SUCCESS
default proxy-admin authorizationpolicy/admin-von-prometheus 0.0rps 100.00%
probe proxy-admin default/probe 0.0rps 100.00%
default web-http authorizationpolicy/web-von-last 0.0rps 89.92%
Status check results are √
Same rates as on Day 0, no more audit, no more
default:all-unauthenticated. In the Grafana dashboard
“Linkerd Deployment” you see a gap of a few minutes (Drill 6), after
which the curves continue. These policies stay active for the rest of
the training.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all policy resources in
noteswith one command. Target time: 20 s - Show which default policy the proxy of
helperhas. Target time: 45 s - Show which authorizations the proxy of
notes-apiknows for port 8080. Target time: 45 s - Prove that
fremdcan no longer get into the database. Target time: 30 s
Solution
kubectl -n notes get server,authorizationpolicy,meshtlsauthentication,\
networkauthentication
kubectl -n notes get pod -l app=helper -o yaml | grep -m1 -A1 INBOUND_DEFAULT_POLICY
P=$(kubectl -n notes get pod -l app=notes-api -o name | head -1)
linkerd diagnostics policy -n notes $P 8080 -o json \
| jq -r '.authorizations[].labels.name'
kubectl -n tag06 exec deploy/fremd -c psql -- psql $DB -tAc 'select 1'Expected (abridged):
NAME PORT PROTOCOL ACCESS POLICY
server.policy.linkerd.io/helper-http http HTTP/1 deny
server.policy.linkerd.io/notes-api-http http HTTP/1 deny
server.policy.linkerd.io/postgres postgres opaque deny
server.policy.linkerd.io/proxy-admin linkerd-admin HTTP/1 deny
server.policy.linkerd.io/web-http http HTTP/1 deny
- name: LINKERD2_PROXY_INBOUND_DEFAULT_POLICY
value: deny
notes-api-von-web
psql: error: connection to server at "postgres.notes" (10.109.208.153), port 5432
failed: server closed the connection unexpectedly
linkerd diagnostics policy wants a Pod or a Service, not
a Deployment.
Save your progress? Sign up for free
Checkpoint
kubectl -n tag06 exec deploy/fremd -c curl -- curl -s -o /dev/null \
-w '%{http_code}\n' http://helper.notes/getExpected: 403
kubectl -n notes get server -o custom-columns=NAME:.metadata.name,\
POLICY:.spec.accessPolicyExpected: five Servers, all deny (with
no value set, proxy-admin defaults to
deny).
kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -cExpected: /api/notes only
200, /flaky about 70 percent
200.
Challenge
In tag06, create a copy of helper
(ServiceAccount helper, one replica) and allow
fremd only GET /get there. The readiness probe
must not break.
Hint: As soon as a route is attached to a Server, Linkerd no longer allows the probes by itself – and the kubelet has no identity.
Solution
kubectl -n tag06 create serviceaccount helper
sed -e 's/namespace: notes/namespace: tag06/' -e 's/replicas: 2/replicas: 1/' \
~/linkerd/tag00/helper.yaml > helper-kopie.yaml
kubectl apply -f helper-kopie.yaml
kubectl -n tag06 set serviceaccount deploy/helper helper
kubectl -n tag06 rollout status deploy/helper --timeout=120s
cat > challenge.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: helper-http, namespace: tag06}
spec:
podSelector: {matchLabels: {app: helper}}
port: http
proxyProtocol: HTTP/1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: helper-get, namespace: tag06}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: Server, name: helper-http}
rules:
- matches:
- path: {type: Exact, value: /get}
method: GET
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: fremd, namespace: tag06}
spec:
identityRefs: [{kind: ServiceAccount, name: fremd}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-get-von-fremd, namespace: tag06}
spec:
targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: helper-get}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: fremd}
EOF
kubectl apply -f challenge.yaml
sleep 45
kubectl -n tag06 get pods -l app=helper
kubectl -n tag06 get events --field-selector reason=Unhealthy | tail -1Expected (abridged):
NAME READY STATUS RESTARTS AGE
helper-79d57fbbbb-xx9jg 1/2 Running 0 54s
8s Warning Unhealthy pod/helper-79d57fbbbb-xx9jg Readiness probe failed: HTTP
probe failed with statuscode: 404
The probe /status/200 no longer matches any route: 404.
So you need a route for the probe and a rule for the kubelet. Where it
comes from is shown in the log as soon as the route exists:
cat > probe-route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: helper-probe, namespace: tag06}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: Server, name: helper-http}
rules:
- matches:
- path: {type: Exact, value: /status/200}
method: GET
EOF
kubectl apply -f probe-route.yaml
sleep 25
kubectl -n tag06 logs deploy/helper -c linkerd-proxy --since=20s \
| grep 'Request denied' | grep -oE 'route.name=[^ ]*|client.ip=[0-9.]*' | sort -u
kubectl get ciliumnodes \
-o jsonpath='{..addresses[?(@.type=="CiliumInternalIP")].ip}{"\n"}'Expected:
client.ip=10.0.1.116
route.name=helper-probe
10.0.0.88 10.0.1.116 10.0.2.241
With Cilium, the kubelet comes in via the node’s
cilium_host address. You enter those (your IPs will
differ):
cat > kubelet.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: kubelet, namespace: tag06}
spec:
networks:
- cidr: 10.0.0.88/32
- cidr: 10.0.1.116/32
- cidr: 10.0.2.241/32
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-probe-von-kubelet, namespace: tag06}
spec:
targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: helper-probe}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: NetworkAuthentication, name: kubelet}
EOF
kubectl apply -f kubelet.yaml
sleep 25
kubectl -n tag06 get pods -l app=helper
for u in /get /status/200 /headers; do
kubectl -n tag06 exec deploy/fremd -c curl -- curl -s -o /dev/null \
-w "$u %{http_code}\n" http://helper.tag06$u
doneExpected:
NAME READY STATUS RESTARTS AGE
helper-79d57fbbbb-xx9jg 2/2 Running 0 2m3s
/get 200
/status/200 403
/headers 404
/get is allowed, the probe only for the kubelet,
everything else matches no route (404).
Save your progress? Sign up for free
Cleanup
kubectl delete namespace tag06The policies in notes stay.
Interview Note
- How do you introduce authorization without an outage?
Servers with
accessPolicy: audit, create rules, evaluate the audit by client identity, thendeny. Finally, set the namespace’s default policy todenyfor new ports. - What happens to probes and Prometheus? Linkerd allows probes automatically as long as the Server has no routes. Prometheus without a proxy needs a NetworkAuthentication for the admin port 4191.
- 403 or a dropped connection? HTTP gets a 403 from the proxy, opaque/TCP is closed.
Enterprise Note
- Policies belong in Git next to the Deployments; every new edge is a review.
- Set the default policy cluster-wide via Helm
(
proxy.defaultInboundPolicy) instead of per namespace – but it only takes effect after all Pods restart. - Bring Prometheus into the mesh and allow it by identity instead of opening up the Pod network.
- Alert on
inbound_http_authz_deny_totalandinbound_tcp_authz_deny_total: a rise is either an attack or a forgotten legitimate client. - Run jobs, migrations and backups with a native sidecar; otherwise they hit a wall under default deny.