Day 16: Troubleshooting
Goal: You find the typical mesh faults with the Linkerd tools in a few minutes and fix them without guessing. · Duration: approx. 90 min · Prerequisite: Day 7
Briefing
- Work from the outside in: load generator,
linkerd check, metrics, the proxy’s policy, and only then logs. - Today you build in every fault yourself – in a copy of the app in
the namespace
tag16with default deny, just like innotes. - For each fault: build it in, look at the symptom, prove the cause with tools, fix it, measure again.
linkerd diagnosticsqueries the proxy or the Control Plane directly and needs no Prometheus. That helps exactly when viz itself is blind.
Approach
| Symptom | First command | Common cause |
|---|---|---|
Pod 1/1 instead of 2/2 |
linkerd check --proxy -n <ns> |
not restarted |
Everything 403 |
Proxy log grep 'Request denied' |
no Server/no rule |
Single client 403 |
linkerd viz edges -o wide |
wrong ServiceAccount |
| Exactly 10 s wait | linkerd diagnostics policy svc/<name> <port> |
opaque port missing |
Init:Error, DB closed |
proxy-metrics for port_skipped |
skip port, old sidecar |
| Retry/timeout has no effect | linkerd diagnostics policy svc/<name> <port> |
wrong parentRef |
503 from client proxy |
linkerd diagnostics endpoints <fqdn>:<port> |
no endpoints |
| Proxy restarts | kubectl get pod -o jsonpath=…lastState |
OOMKilled |
| Pod not Ready | kubectl get events |
route on the Server (Day 6) |
| TLS errors, no identity | linkerd check (identity section) |
issuer expired, clock |
Drill 1: Annotated, but not in the mesh
The copy of the app: one replica per service, dedicated ServiceAccounts, Postgres with a volume. The namespace is annotated only after the deployment – as so often happens in operations.
mkdir -p ~/linkerd/tag16 && cd ~/linkerd/tag16
kubectl create namespace tag16
for a in web notes-api helper last postgres; do
kubectl -n tag16 create serviceaccount $a
done
cat > app.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata: {name: notes-db, namespace: tag16}
stringData:
password: lab-db-123
url: postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: postgres, namespace: tag16}
spec:
serviceName: postgres
replicas: 1
selector: {matchLabels: {app: postgres}}
template:
metadata: {labels: {app: postgres}}
spec:
serviceAccountName: postgres
containers:
- name: postgres
image: docker.io/library/postgres:18.7-alpine
env:
- {name: POSTGRES_USER, value: notes}
- {name: POSTGRES_DB, value: notes}
- name: POSTGRES_PASSWORD
valueFrom: {secretKeyRef: {name: notes-db, key: password}}
- {name: PGDATA, value: /var/lib/postgresql/data/pgdata}
ports: [{containerPort: 5432, name: postgres}]
readinessProbe:
exec: {command: [pg_isready, -U, notes, -d, notes]}
volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
volumeClaimTemplates:
- metadata: {name: data}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: tag16
annotations: {config.linkerd.io/opaque-ports: "5432"}
spec:
selector: {app: postgres}
ports: [{port: 5432, targetPort: postgres, name: postgres}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: tag16}
spec:
replicas: 1
selector: {matchLabels: {app: notes-api}}
template:
metadata: {labels: {app: notes-api}}
spec:
serviceAccountName: notes-api
initContainers:
- name: migrate
image: ghcr.io/thomaszachmann/notes-api:0.7.1
args: [migrate]
env:
- name: NOTES_DB_URL
valueFrom: {secretKeyRef: {name: notes-db, key: url}}
containers:
- name: notes-api
image: ghcr.io/thomaszachmann/notes-api:0.7.1
ports: [{containerPort: 8080, name: http}]
env:
- name: NOTES_DB_URL
valueFrom: {secretKeyRef: {name: notes-db, key: url}}
- {name: NOTES_API_KEY, value: lab-key-123}
readinessProbe:
httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag16}
spec:
selector: {app: notes-api}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: tag16}
spec:
replicas: 1
selector: {matchLabels: {app: helper}}
template:
metadata: {labels: {app: helper}}
spec:
serviceAccountName: helper
containers:
- name: helper
image: ghcr.io/mccutchen/go-httpbin:2.26.0
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: tag16}
spec:
selector: {app: helper}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag16}
data:
default.conf: |
server {
listen 8080;
location /api/ {
proxy_set_header X-API-Key lab-key-123;
proxy_pass http://notes-api.tag16.svc.cluster.local/;
}
location /flaky {
proxy_pass http://helper.tag16.svc.cluster.local/unstable?failure_rate=0.3;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag16}
spec:
replicas: 1
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
serviceAccountName: web
containers:
- name: web
image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
ports: [{containerPort: 8080, name: http}]
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag16}
spec:
selector: {app: web}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag16}
spec:
replicas: 1
selector: {matchLabels: {app: last}}
template:
metadata: {labels: {app: last}}
spec:
serviceAccountName: last
containers:
- name: last
image: docker.io/curlimages/curl:8.23.0
command: [sh, -c]
args:
- |
while true; do
for p in /api/notes /api/notes /flaky; do
code=$(curl -s -o /dev/null -w '%{http_code}' -m 5 http://web$p)
echo "$(date +%T) $p $code"
done
sleep 1
done
EOF
kubectl apply -f app.yaml
kubectl -n tag16 rollout status statefulset/postgres --timeout=180s
kubectl -n tag16 rollout status deploy/notes-api --timeout=180s
kubectl annotate namespace tag16 linkerd.io/inject=enabled \
config.linkerd.io/default-inbound-policy=denyThe app is running, and last measures 100 percent for
/api/notes. But is it in the mesh?
kubectl -n tag16 get pods -o custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd viz stat -n tag16 deploy
linkerd check --proxy -n tag16 --wait 30s 2>&1 | grep -A2 'proxies are ready'Expected (abridged):
NAME PROXY
helper-79d57fbbbb-mqhpg <none>
last-78895758f9-52xd7 <none>
notes-api-5bc98b7fb-n4s5j <none>
postgres-0 <none>
web-7d94ccc84d-qjrst <none>
No traffic found.
× data plane proxies are ready
no "linkerd-proxy" containers found in the "tag16" namespace
see https://linkerd.io/2/checks/#l5d-data-plane-ready for hints
The webhook only kicks in when a Pod is created (Day 2).
--wait 30s matters: without it, check --proxy
waits up to five minutes for proxies to show up. The fix is a
restart:
kubectl -n tag16 rollout restart deploy
kubectl -n tag16 rollout restart statefulset/postgres
kubectl -n tag16 rollout status statefulset/postgres --timeout=180s
sleep 45
kubectl -n tag16 get pods
kubectl -n tag16 logs deploy/last -c last --since=20s | awk '{print $2, $3}' \
| sort | uniq -cExpected (abridged):
NAME READY STATUS RESTARTS AGE
helper-5c9445c87c-khfww 2/2 Running 0 53s
last-6ddfdfcc46-8cpmp 2/2 Running 0 53s
notes-api-5b6fbd4d9c-fd4n7 1/2 Init:Error 2 (38s ago) 52s
notes-api-5bc98b7fb-n4s5j 0/1 Running 0 2m3s
postgres-0 2/2 Running 0 51s
web-6df978d4f-mhgj5 2/2 Running 0 52s
30 /api/notes 403
14 /flaky 403
Everything is in the mesh – and everything is broken. That is the next fault.
Save your progress? Sign up for free
Drill 2: Default deny without rules
With the restart, the default policy deny from the
namespace annotation applies. What does viz show?
linkerd viz stat -n tag16 deploy
linkerd viz authz -n tag16 deploy/webExpected:
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 TCP_CONN
helper 1/1 - - - - - -
last 1/1 - - - - - -
notes-api 1/2 - - - - - -
web 1/1 - - - - - -
ROUTE SERVER AUTHORIZATION UNAUTHORIZED SUCCESS RPS LATENCY_P50 LATENCY_P95 ...
Nothing. viz reads Prometheus, and Prometheus itself is rejected on
port 4191 (Day 6, Drill 6). So go straight to the proxy – via the log
and diagnostics, which arrive over localhost
through a port forward:
kubectl -n tag16 logs deploy/web -c linkerd-proxy --since=1m \
| grep 'Request denied' | grep -v admin | tail -1 \
| grep -oE 'server.name=[^ ]*|client.tls=[^ ]*'
P=$(kubectl -n tag16 get pod -l app=web -o name)
linkerd diagnostics policy -n tag16 $P 8080 -o json \
| jq -c '{server: .labels.name, authz: .authorizations}'
linkerd diagnostics proxy-metrics -n tag16 po/postgres-0 \
| grep '^inbound_tcp_authz_deny_total' \
| grep -oE '(srv_name|client_id)="[^"]*"|\} [0-9]+' | paste - - -Expected (abridged):
server.name=deny
client.tls=Some(Established
{"server":"deny","authz":null}
srv_name="deny" client_id="notes-api.tag16.serviceaccount.identity.linkerd.cl... } 7
srv_name="deny" } 34
server.name=deny means: no Server matches, the default
policy applies, and it contains not a single authorization. That is why
Postgres also rejects migrate (Init:Error).
The rules from Day 6 fit one to one, only the namespace differs:
for f in servers authz admin; do
sed 's/namespace: notes/namespace: tag16/' ~/linkerd/tag06/$f.yaml; echo ---
done > policy.yaml
grep -c '^kind:' policy.yaml
kubectl apply -f policy.yaml
kubectl -n tag16 rollout status deploy/notes-api --timeout=180s
sleep 60
kubectl -n tag16 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n tag16 deployExpected (abridged, numbers vary):
14
...
deployment "notes-api" successfully rolled out
80 /api/notes 200
4 /api/notes 503
29 /flaky 200
12 /flaky 500
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 ...
helper 1/1 80.88% 1.1rps 1ms 14ms 19ms
last 1/1 100.00% 0.3rps 1ms 49ms 50ms
notes-api 1/1 100.00% 1.7rps 7ms 35ms 39ms
web 1/1 87.42% 2.5rps 16ms 69ms 94ms
The four 503s fall into the seconds before
migrate starts again after its backoff: without a ready
Pod, the proxy of web answers with 503 itself after three
seconds. From now on, tag16 is healthy – the baseline for
the next faults.
Warning:
linkerd viz check --proxywarns here (and innotes) with “prometheus is authorized to scrape data plane pods”. The check looks for a rule for viz’s own Prometheus. With the NetworkAuthentication from Day 6, this is a false alarm – rely onupin Prometheus.
Save your progress? Sign up for free
Drill 3: The wrong ServiceAccount
A colleague redeploys last and forgets
serviceAccountName:
kubectl -n tag16 set serviceaccount deploy/last default
kubectl -n tag16 rollout status deploy/last --timeout=120s
sleep 60
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz authz -n tag16 deploy/web | grep web-http
linkerd viz edges -n tag16 deploy -o wideExpected (abridged):
62 /api/notes 403
30 /flaky 403
default web-http 2.3rps 0.00% 0.0rps ...
default web-http authorizationpolicy/web-von-last 0.0rps 89.71% 1.1rps ...
SRC DST SRC_NS DST_NS CLIENT_ID SERVER_ID SECURED
last web tag16 tag16 default.tag16 web.tag16 √
The row without a rule showing 2.3 rps in UNAUTHORIZED
is the hit, and edges -o wide names the identity:
default.tag16. The rule, however, expects
last:
kubectl -n tag16 logs deploy/web -c linkerd-proxy --since=1m \
| grep -m1 'Request denied' | grep -oE 'server.name=[^ ]*|Name\("[^"]*'
kubectl -n tag16 get meshtlsauthentication last -o jsonpath='{.spec}{"\n"}'Expected:
server.name=web-http
Name("default.tag16.serviceaccount.identity.linkerd.cluster.local
{"identityRefs":[{"kind":"ServiceAccount","name":"last"}]}
Tip:
linkerd viz tapdoes not show rejected requests – they end before the point where tap listens in. For 403s, the proxy log andviz authzare the tools.
You can check which identity a Pod speaks with even without a shell
in the image: an ephemeral container shares the Pod’s network and proxy.
From inside helper, which is not allowed to call
notes-api:
P=$(kubectl -n tag16 get pod -l app=helper -o jsonpath='{.items[0].metadata.name}')
kubectl -n tag16 debug $P -q --image=docker.io/curlimages/curl:8.23.0 -c probe -- \
curl -s -o /dev/null -w '%{http_code}\n' -H 'X-API-Key: lab-key-123' \
http://notes-api/notes
sleep 5
kubectl -n tag16 logs $P -c probe
kubectl -n tag16 logs deploy/notes-api -c linkerd-proxy --since=1m \
| grep 'Request denied' | tail -1 | grep -oE 'Name\("[^"]*'Expected:
403
Name("helper.tag16.serviceaccount.identity.linkerd.cluster.local
The debug container has the Pod’s identity – handy for testing a rule
from the point of view of exactly this client. Now repair
last:
kubectl -n tag16 set serviceaccount deploy/last last
kubectl -n tag16 rollout status deploy/last --timeout=120s
sleep 45
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -cExpected (numbers vary):
54 /api/notes 200
21 /flaky 200
6 /flaky 500
You get the same picture when the MeshTLSAuthentication contains a
typo or the ServiceAccount comes from a different namespace
(identityRefs with namespace).
Save your progress? Sign up for free
Drill 4: Ten seconds of silence
A service where the server speaks first – like SMTP or MySQL. It announces itself with a greeting line on port 2525, which is not on Linkerd’s opaque list:
kubectl -n tag16 create serviceaccount banner
cat > banner.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: banner, namespace: tag16}
spec:
replicas: 1
selector: {matchLabels: {app: banner}}
template:
metadata: {labels: {app: banner}}
spec:
serviceAccountName: banner
containers:
- name: banner
image: docker.io/library/busybox:1.37
command: [nc, -lk, -p, "2525", -e, echo, "220 banner bereit"]
ports: [{containerPort: 2525, name: smtp}]
---
apiVersion: v1
kind: Service
metadata: {name: banner, namespace: tag16}
spec:
selector: {app: banner}
ports: [{port: 2525, targetPort: smtp, name: smtp}]
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: banner, namespace: tag16}
spec:
podSelector: {matchLabels: {app: banner}}
port: smtp
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: banner-von-last, namespace: tag16}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: banner}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: last}
EOF
kubectl apply -f banner.yaml
kubectl -n tag16 rollout status deploy/banner --timeout=120s
kubectl -n tag16 exec deploy/last -c last -- sh -c 's=$(date +%s); (sleep 15) \
| nc banner 2525 | while read l; do echo "$(( $(date +%s) - s )) s: $l"; done'Expected:
10 s: 220 banner bereit
Exactly ten seconds. The client proxy waits for the client’s first bytes to detect the protocol. The client waits for the server. Both wait until the proxy gives up:
kubectl -n tag16 logs deploy/last -c linkerd-proxy --since=1m | grep 'after timeout'
linkerd diagnostics policy -n tag16 svc/banner 2525 -o json \
| jq -c '.protocol.Kind | keys, .Detect.timeout'Expected (abridged):
INFO ThreadId(01) outbound:proxy{addr=10.99.111.91:2525}: linkerd_app_outbound::pro...
...tocol: Continuing after timeout: 10s
["Detect"]
{"seconds":10}
diagnostics policy with a Service shows
the outbound view: protocol Detect with a ten-second
deadline. So: opaque port, on the Service:
kubectl -n tag16 annotate service banner config.linkerd.io/opaque-ports=2525
sleep 5
linkerd diagnostics policy -n tag16 svc/banner 2525 -o json \
| jq -c '.protocol.Kind | keys'
kubectl -n tag16 exec deploy/last -c last -- sh -c 's=$(date +%s); (sleep 3) \
| nc banner 2525 | while read l; do echo "$(( $(date +%s) - s )) s: $l"; done'
linkerd check --proxy -n tag16 --wait 30s 2>&1 | grep -A2 opaqueExpected (abridged):
["Opaque"]
0 s: 220 banner bereit
‼ opaque ports are properly annotated
* service banner expects target port 2525 to be opaque; add it to pod
banner-84d6cdb7b8-pl2kb config.linkerd.io/opaque-ports annotation
For clients in the mesh, the Service is enough. check
also wants the annotation on the Pod – for direct Pod calls and headless
Services:
kubectl -n tag16 patch deploy banner -p '{"spec":{"template":{"metadata":
{"annotations":{"config.linkerd.io/opaque-ports":"2525"}}}}}'
kubectl -n tag16 rollout status deploy/banner --timeout=120s
linkerd check --proxy -n tag16 --wait 30s 2>&1 | grep -E 'opaque|Status check'Expected:
√ opaque ports are properly annotated
Status check results are √
Warning: The annotation
config.linkerd.io/opaque-portson the workload replaces the default list (25, 587, 3306, …) for this Pod instead of extending it. As an alternative on the Service, you can useappProtocol: linkerd.io/opaqueon the port.
Save your progress? Sign up for free
Drill 5: Bypassing the proxy
Someone wanted to make the database connection “faster” and takes port 5432 out of the proxy:
kubectl -n tag16 patch deploy notes-api -p '{"spec":{"template":{"metadata":
{"annotations":{"config.linkerd.io/skip-outbound-ports":"5432"}}}}}'
sleep 50
kubectl -n tag16 get pods -l app=notes-apiExpected:
NAME READY STATUS RESTARTS AGE
notes-api-5b6fbd4d9c-fd4n7 2/2 Running 0 9m37s
notes-api-794d4966dd-bzmxp 1/2 Init:Error 2 (42s ago) 50s
The old Pod keeps running, users notice nothing – the rollout is stuck. Who is rejecting?
P=$(kubectl -n tag16 get pod -l app=notes-api \
--sort-by=.metadata.creationTimestamp -o name | tail -1)
kubectl -n tag16 logs $P -c migrate --tail=1
linkerd diagnostics proxy-metrics -n tag16 po/postgres-0 \
| grep '^inbound_tcp_authz_deny_total' \
| grep -oE '(srv_name|no_tls_reason)="[^"]*"|\} [0-9]+' | paste - - -
kubectl -n tag16 get $P -o jsonpath='{.spec.initContainers[0].args}' \
| grep -o '"--outbound-ports-to-ignore","[^"]*"'Expected (abridged):
time=2026-11-19T20:05:56.868Z level=ERROR msg="database unreachable" source=env
err="failed to connect to `user=notes database=notes`: 10.100.223.86:5432 (postgres):
failed to receive message: unexpected EOF"
srv_name="postgres" no_tls_reason="port_skipped" } 16
"--outbound-ports-to-ignore","5432"
port_skipped means: plaintext on an opaque port.
linkerd-init routed 5432 past the iptables rules, so
notes-api talks without a proxy and without an identity.
Revert it:
kubectl -n tag16 patch deploy notes-api --type=json -p '[{"op":"remove",
"path":"/spec/template/metadata/annotations/config.linkerd.io~1skip-outbound-ports"}]'
kubectl -n tag16 rollout status deploy/notes-api --timeout=180s
kubectl -n tag16 get pods -l app=notes-apiExpected (abridged):
deployment "notes-api" successfully rolled out
NAME READY STATUS RESTARTS AGE
notes-api-5b6fbd4d9c-fd4n7 2/2 Running 0 9m48s
notes-api-794d4966dd-bzmxp 1/2 Terminating 3 (38s ago) 61s
An init container without a native sidecar (Day 6, Drill 7) produces
exactly the same picture – Init:Error,
port_skipped. The telltale signs there: the annotation
config.beta.linkerd.io/proxy-enable-native-sidecar: "false"
and linkerd-init after your init container
in spec.initContainers.
Tip: The arguments of
linkerd-initalso contain the default--inbound-ports-to-ignore 4190,4191,4567,4568. So Linkerd already accepts nothing inbound on 4567/4568 (Galera) on its own.
Save your progress? Sign up for free
Drill 6: The route that does nothing
Retries for helper as on Day 7 – with a small mistake in
the parentRef:
cat > route-retry.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: helper-retry
namespace: tag16
annotations:
retry.linkerd.io/http: 5xx
retry.linkerd.io/limit: "2"
spec:
parentRefs:
- {group: core, kind: Service, name: helper, port: 8080}
rules:
- matches:
- path: {type: PathPrefix, value: /}
method: GET
EOF
kubectl apply -f route-retry.yaml
sleep 65
kubectl -n tag16 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag16 get httproute helper-retry \
-o jsonpath='{.status.parents[0].conditions[*].reason}{"\n"}'Expected (numbers vary):
92 /api/notes 200
33 /flaky 200
13 /flaky 500
Accepted ResolvedRefs
Still 70 percent, even though the route is Accepted. The
policy controller does not check whether the Service even has port 8080.
What the proxy actually receives:
linkerd diagnostics policy -n tag16 svc/helper 80 -o json \
| jq -c '.protocol.Kind.Detect.http1.routes[].metadata.Kind'
linkerd diagnostics policy -n tag16 svc/helper 8080 -o json \
| jq -c '.protocol.Kind.Detect.http1.routes[].metadata.Kind'
linkerd diagnostics proxy-metrics -n tag16 deploy/web \
| grep -c '^outbound_http_route_retry_requests_total'Expected:
{"Default":"http"}
{"Resource":{"group":"gateway.networking.k8s.io","kind":"HTTPRoute","name":"helper-r...
0
parentRef.port is the Service port
(80), not the container port (8080). On port 80, the proxy sees only the
default route, zero retries.
sed -i 's/port: 8080}/port: 80}/' route-retry.yaml
kubectl apply -f route-retry.yaml
sleep 65
kubectl -n tag16 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd diagnostics proxy-metrics -n tag16 deploy/web \
| grep '^outbound_http_route_retry_requests_total' \
| grep -oE 'route_name="[^"]*"|[0-9]+$' | paste - -Expected (numbers vary):
84 /api/notes 200
40 /flaky 200
2 /flaky 500
route_name="helper-retry" 14
Other candidates when a route does nothing: a forgotten
group (if omitted, gateway.networking.k8s.io
applies), a route in a different namespace than the Service, a
ServiceProfile for the same Service (it wins, Day 7), or a client
without a proxy – only the client proxy performs retries. A retry budget
as in the old ServiceProfile does not exist on HTTPRoutes, only
limit.
Save your progress? Sign up for free
Drill 7: A renamed port
A developer renames the container port in helper from
http to web and updates the probe
accordingly:
kubectl -n tag16 patch deploy helper --type=json -p '[
{"op":"replace","path":"/spec/template/spec/containers/0/ports/0/name",
"value":"web"},
{"op":"replace",
"path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port",
"value":"web"}]'
kubectl -n tag16 rollout status deploy/helper --timeout=120s
sleep 50
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80
kubectl -n tag16 get endpointslices -l kubernetes.io/service-name=helperExpected:
58 /api/notes 200
29 /flaky 503
No endpoints found.
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
helper-jcdjm IPv4 <unset> 10.0.1.106 60s
The Pod is ready, but the Service looks for
targetPort: http – which no longer exists.
diagnostics endpoints shows what
linkerd-destination hands out to the proxies: nothing. For
more detail in the proxy of web, raise the log level. The
annotation only takes effect after a restart and produces a lot of text
– use it only briefly:
kubectl -n tag16 patch deploy web -p '{"spec":{"template":{"metadata":
{"annotations":{"config.linkerd.io/proxy-log-level":"warn,linkerd=debug"}}}}}'
kubectl -n tag16 rollout status deploy/web --timeout=120s
sleep 30
kubectl -n tag16 logs deploy/web -c linkerd-proxy --since=30s \
| grep -oE 'name=helper port=80\}:pool: .*' | cut -c1-80 | sort | uniq -c \
| sort -rn | head -4Expected (abridged):
21 name=helper port=80}:pool: linkerd_pool_p2c: No ready endpoints
7 name=helper port=80}:pool: linkerd_proxy_api_resolve::resolve: Resolving ds
6 name=helper port=80}:pool: linkerd_proxy_resolve::recover: recovering err=c
1 name=helper port=80}:pool: linkerd_proxy_balance_queue::worker: Unavailable
“No ready endpoints”, then fail-fast after three seconds: hence the 503s. The obvious repair is to update the Service to match:
kubectl -n tag16 patch service helper --type=json \
-p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":"web"}]'
sleep 20
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80
kubectl -n tag16 logs deploy/last -c last --since=12s | awk '{print $2, $3}' \
| sort | uniq -c
P=$(kubectl -n tag16 get pod -l app=helper -o name)
linkerd diagnostics policy -n tag16 $P 8080 -o json | jq -c '{server: .labels.name}'Expected:
NAMESPACE IP PORT POD SERVICE
tag16 10.0.1.106 8080 helper-9d8c45bb8-gsbm5 helper.tag16
18 /api/notes 200
9 /flaky 403
{"server":"deny"}
Endpoints are there, now 403: the Server helper-http
also looks for port: http. The port falls under the default
policy. Port names are a contract between Pod, Service, and Server. The
right fix is to rename the port back – and revert the log level at the
same time:
kubectl -n tag16 patch deploy helper --type=json -p '[
{"op":"replace","path":"/spec/template/spec/containers/0/ports/0/name",
"value":"http"},
{"op":"replace",
"path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port",
"value":"http"}]'
kubectl -n tag16 patch service helper --type=json \
-p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":"http"}]'
kubectl -n tag16 patch deploy web --type=json -p '[{"op":"remove",
"path":"/spec/template/metadata/annotations/config.linkerd.io~1proxy-log-level"}]'
kubectl -n tag16 rollout status deploy/helper --timeout=120s
kubectl -n tag16 rollout status deploy/web --timeout=120s
sleep 50
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -c
P=$(kubectl -n tag16 get pod -l app=helper -o name)
linkerd diagnostics policy -n tag16 $P 8080 -o json | jq -c '{server: .labels.name}'Expected (numbers vary):
59 /api/notes 200
28 /flaky 200
1 /flaky 500
{"server":"helper-http"}
Save your progress? Sign up for free
Drill 8: Proxy OOMKilled
You set resources for the proxy via annotation. A limit that is too tight goes unnoticed while idle – the proxy here needs only about 3 MiB:
kubectl top pod -n tag16 --containers | grep -E 'POD|helper.*proxy'
kubectl -n tag16 patch deploy helper -p '{"spec":{"template":{"metadata":
{"annotations":{"config.linkerd.io/proxy-memory-limit":"6Mi"}}}}}'
kubectl -n tag16 rollout status deploy/helper --timeout=120s
kubectl -n tag16 get pods -l app=helperExpected (abridged):
POD NAME CPU(cores) MEMORY(bytes)
helper-5c9445c87c-khfww linkerd-proxy 4m 3Mi
NAME READY STATUS RESTARTS AGE
helper-665b47786f-lt26b 2/2 Running 0 45s
It runs. Then load arrives: 200 concurrent, slow requests from
web (go-httpbin /drip trickles the response
over five seconds):
kubectl -n tag16 exec deploy/web -c web -- sh -c 'seq 400 | xargs -P 200 -I{} \
curl -s -o /dev/null -w "%{http_code}\n" -m 20 \
"http://helper/drip?duration=5&numbytes=5&delay=0"' | sort | uniq -c
kubectl -n tag16 get pods -l app=helper
kubectl -n tag16 get pod -l app=helper -o custom-columns=NAME:.metadata.name,\
PROXY-RESTARTS:.status.initContainerStatuses[1].restartCount,\
GRUND:.status.initContainerStatuses[1].lastState.terminated.reason
kubectl -n tag16 get pod -l app=helper \
-o jsonpath='{.items[0].spec.initContainers[1].resources}{"\n"}'Expected (abridged, numbers vary):
command terminated with exit code 123
199 200
201 503
NAME READY STATUS RESTARTS AGE
helper-665b47786f-5hwgq 2/2 Running 1 (11s ago) 30s
NAME PROXY-RESTARTS GRUND
helper-665b47786f-5hwgq 1 OOMKilled
{"limits":{"memory":"6Mi"},"requests":{"memory":"6Mi"}}
Exit code 123 is reported only by xargs, because
individual curl calls failed. The proxy is a long-running
init container, which is why the reason appears under
initContainerStatuses. If you catch the Pod in backoff, its
status reads Init:CrashLoopBackOff. While the proxy
restarts, helper is gone: 503 from the proxy in
web. Afterwards the Pod comes back up – until the next load
spike. Without a request annotation, Kubernetes sets the
request equal to the limit. Set limits after measuring under load, with
headroom:
kubectl -n tag16 patch deploy helper -p '{"spec":{"template":{"metadata":
{"annotations":{"config.linkerd.io/proxy-memory-limit":"128Mi",
"config.linkerd.io/proxy-memory-request":"20Mi"}}}}}'
kubectl -n tag16 rollout status deploy/helper --timeout=120s
kubectl -n tag16 exec deploy/web -c web -- sh -c 'seq 400 | xargs -P 200 -I{} \
curl -s -o /dev/null -w "%{http_code}\n" -m 20 \
"http://helper/drip?duration=5&numbytes=5&delay=0"' | sort | uniq -c
kubectl -n tag16 get pods -l app=helper
kubectl top pod -n tag16 --containers | grep helperExpected (abridged):
400 200
NAME READY STATUS RESTARTS AGE
helper-5bf6f997c8-zdqlt 2/2 Running 0 26s
helper-5bf6f997c8-zdqlt helper 58m 9Mi
helper-5bf6f997c8-zdqlt linkerd-proxy 63m 7Mi
The proxy’s memory grows with concurrent connections and requests, not with the number of Pods in the cluster.
Tip: If the
proxy-memory-requestis greater than the limit, creating the Pod already fails (FailedCreate, “must be less than or equal to memory limit” – also forlinkerd-init).rollout statusthen just waits silently; the cause is inkubectl get events.
Warning: Better not reproduce certificate and clock errors – they hit the entire mesh. If the issuer expires, new proxies no longer get an identity (
linkerd checkreports it underlinkerd-identity, the proxy log shows errors fetching the certificate, Pods do not become Ready). If a node’s clock is wrong, certificates there are “not yet” or “no longer” valid: mTLS handshakes fail only from and to Pods on that node. Early warning:control_identity_cert_refreshes_total{result="error"}(Day 3), expiry dates of issuer and anchor (Day 5), and NTP on all nodes.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show which Pods in
tag16have a proxy and which version. Target time: 20 s - Show which Server and which rules apply to port 5432 in
postgres-0. Target time: 40 s - Show which identities
notes-apihas rejected so far and how often. Target time: 60 s - Show when the issuer and the proxy certificate of
webexpire. Target time: 40 s
Solution
kubectl -n tag16 get pods -o custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd diagnostics policy -n tag16 po/postgres-0 5432 -o json \
| jq -c '{server: .labels.name, authz: [.authorizations[].labels.name]}'
linkerd diagnostics proxy-metrics -n tag16 deploy/notes-api \
| grep '^inbound_http_authz_deny_total' \
| sed -E 's/.*client_id="([^"]*)".*\} /\1 /; s/^inbound.*\} /ohne-identitaet /'
kubectl -n linkerd get certificate linkerd-identity-issuer \
-o jsonpath='{.status.notAfter}{"\n"}'
linkerd identity -n tag16 -l app=web | grep 'Not After'Expected (abridged):
NAME PROXY
banner-f67576cf9-6kptm edge-26.11.2
...
{"server":"postgres","authz":["postgres-von-notes-api"]}
helper.tag16.serviceaccount.identity.linkerd.cluster.local 1
ohne-identitaet 6
2026-11-21T16:02:00Z
Not After : Nov 20 20:19:15 2026 UTC
ohne-identitaet (without identity) are the Prometheus
scrapes on 4191 from Drill 2, before proxy-admin was in
place. The issuer is valid for 48 hours and is renewed by cert-manager
(Day 5).
Save your progress? Sign up for free
Checkpoint
kubectl -n tag16 get podsExpected: all Pods 2/2 Running.
kubectl -n tag16 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -cExpected: /api/notes only
200, /flaky almost only 200
(retries from Drill 6).
linkerd check --proxy -n tag16 2>&1 | tail -1Expected:
Status check results are √
kubectl -n tag16 get pods -o custom-columns=NAME:.metadata.name,\
PROXY-RESTARTS:.status.initContainerStatuses[1].restartCountExpected: a 0 after every Pod – no
proxy restart since the last rollout.
Challenge
Roll out a “release” that changes two things at once, and then repair everything, using only Linkerd tools and without looking at the commands below again:
kubectl -n tag16 create serviceaccount web-v2
kubectl -n tag16 set serviceaccount deploy/web web-v2
kubectl -n tag16 patch service helper -p '{"spec":{"selector":{"app":"helper-v2"}}}'web should keep web-v2.
Hint: Two paths, two different status codes – two causes.
Solution
sleep 50
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag16 logs deploy/notes-api -c linkerd-proxy --since=1m \
| grep -m1 'Request denied' | grep -oE 'server.name=[^ ]*|Name\("[^"]*'
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80
kubectl -n tag16 get service helper -o jsonpath='{.spec.selector}{"\n"}'Expected:
59 /api/notes 403
29 /flaky 503
server.name=notes-api-http
Name("web-v2.tag16.serviceaccount.identity.linkerd.cluster.local
No endpoints found.
{"app":"helper-v2"}
403 on /api/notes: notes-api
sees the new identity web-v2, but the rule only knows
web. 503 on /flaky: the Service
finds no Pod. The MeshTLSAuthentication web is used by two
rules (notes-api-von-web, helper-von-web) –
one change is enough for both:
kubectl -n tag16 patch meshtlsauthentication web --type=merge -p '{"spec":
{"identityRefs":[{"kind":"ServiceAccount","name":"web"},
{"kind":"ServiceAccount","name":"web-v2"}]}}'
kubectl -n tag16 patch service helper -p '{"spec":{"selector":{"app":"helper"}}}'
sleep 50
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80Expected (numbers vary):
48 /api/notes 200
22 /flaky 200
1 /flaky 500
NAMESPACE IP PORT POD SERVICE
tag16 10.0.1.175 8080 helper-5bf6f997c8-l4fkh helper.tag16
Had you looked only for the 403, /flaky would have
stayed at 503 – and vice versa. With multiple symptoms, trace each one
individually down to its cause.
Save your progress? Sign up for free
Cleanup
kubectl delete namespace tag16
kubectl get endpointslices -A | grep cilium-gateway | grep tag16Expected: no output. Along with the namespace, the
PVC and the EndpointSlice cilium-gateway-helper, which
Cilium created for the route from Drill 6, disappear too.
Interview Note
- How do you approach a mesh outage? From the outside in:
measure the symptom,
linkerd check --proxy, viz metrics, thendiagnostics policy/endpointsas the proxy’s view, and finally logs with a raised log level. - 403 or 503 – where do you look? 403 is policy on the server side (Server, identity, route). 503 from the proxy usually means: no ready endpoints, fail-fast in the client proxy.
- Why does a connection hang for exactly ten seconds? Protocol detection with a protocol in which the server speaks first – set an opaque port.
Enterprise Note
- Runbooks per symptom like the table above, with the exact commands. During an outage, nobody reads the docs.
- Alerts on
inbound_http_authz_deny_total,inbound_tcp_authz_deny_total, proxy restarts (kube_pod_init_container_status_restarts_total), and retry rates. - Never keep debug logging permanently: it costs CPU, memory, and log volume. Set the annotation via a change and remove it again.
- Check port names, ServiceAccounts, and opaque ports with an admission policy (Kyverno, Gatekeeper) – most of today’s faults can be caught before deployment.
- Restrict
kubectl debugvia RBAC: a debug container inherits the Pod’s identity and thus all of its permissions in the mesh.