Skip to content
Hands-on IT IT Drills

Day 16: Troubleshooting

Goal: You find the typical mesh faults with the Linkerd tools in a few minutes and fix them without guessing. · Duration: approx. 90 min · Prerequisite: Day 7

Briefing

  • Work from the outside in: load generator, linkerd check, metrics, the proxy’s policy, and only then logs.
  • Today you build in every fault yourself – in a copy of the app in the namespace tag16 with default deny, just like in notes.
  • For each fault: build it in, look at the symptom, prove the cause with tools, fix it, measure again.
  • linkerd diagnostics queries the proxy or the Control Plane directly and needs no Prometheus. That helps exactly when viz itself is blind.

Approach

Symptom First command Common cause
Pod 1/1 instead of 2/2 linkerd check --proxy -n <ns> not restarted
Everything 403 Proxy log grep 'Request denied' no Server/no rule
Single client 403 linkerd viz edges -o wide wrong ServiceAccount
Exactly 10 s wait linkerd diagnostics policy svc/<name> <port> opaque port missing
Init:Error, DB closed proxy-metrics for port_skipped skip port, old sidecar
Retry/timeout has no effect linkerd diagnostics policy svc/<name> <port> wrong parentRef
503 from client proxy linkerd diagnostics endpoints <fqdn>:<port> no endpoints
Proxy restarts kubectl get pod -o jsonpath=…lastState OOMKilled
Pod not Ready kubectl get events route on the Server (Day 6)
TLS errors, no identity linkerd check (identity section) issuer expired, clock

Drill 1: Annotated, but not in the mesh

The copy of the app: one replica per service, dedicated ServiceAccounts, Postgres with a volume. The namespace is annotated only after the deployment – as so often happens in operations.

mkdir -p ~/linkerd/tag16 && cd ~/linkerd/tag16
kubectl create namespace tag16
for a in web notes-api helper last postgres; do
  kubectl -n tag16 create serviceaccount $a
done
cat > app.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata: {name: notes-db, namespace: tag16}
stringData:
  password: lab-db-123
  url: postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: postgres, namespace: tag16}
spec:
  serviceName: postgres
  replicas: 1
  selector: {matchLabels: {app: postgres}}
  template:
    metadata: {labels: {app: postgres}}
    spec:
      serviceAccountName: postgres
      containers:
      - name: postgres
        image: docker.io/library/postgres:18.7-alpine
        env:
        - {name: POSTGRES_USER, value: notes}
        - {name: POSTGRES_DB, value: notes}
        - name: POSTGRES_PASSWORD
          valueFrom: {secretKeyRef: {name: notes-db, key: password}}
        - {name: PGDATA, value: /var/lib/postgresql/data/pgdata}
        ports: [{containerPort: 5432, name: postgres}]
        readinessProbe:
          exec: {command: [pg_isready, -U, notes, -d, notes]}
        volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
  volumeClaimTemplates:
  - metadata: {name: data}
    spec:
      accessModes: [ReadWriteOnce]
      resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: tag16
  annotations: {config.linkerd.io/opaque-ports: "5432"}
spec:
  selector: {app: postgres}
  ports: [{port: 5432, targetPort: postgres, name: postgres}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: tag16}
spec:
  replicas: 1
  selector: {matchLabels: {app: notes-api}}
  template:
    metadata: {labels: {app: notes-api}}
    spec:
      serviceAccountName: notes-api
      initContainers:
      - name: migrate
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        args: [migrate]
        env:
        - name: NOTES_DB_URL
          valueFrom: {secretKeyRef: {name: notes-db, key: url}}
      containers:
      - name: notes-api
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        ports: [{containerPort: 8080, name: http}]
        env:
        - name: NOTES_DB_URL
          valueFrom: {secretKeyRef: {name: notes-db, key: url}}
        - {name: NOTES_API_KEY, value: lab-key-123}
        readinessProbe:
          httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag16}
spec:
  selector: {app: notes-api}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: tag16}
spec:
  replicas: 1
  selector: {matchLabels: {app: helper}}
  template:
    metadata: {labels: {app: helper}}
    spec:
      serviceAccountName: helper
      containers:
      - name: helper
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: tag16}
spec:
  selector: {app: helper}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag16}
data:
  default.conf: |
    server {
      listen 8080;
      location /api/ {
        proxy_set_header X-API-Key lab-key-123;
        proxy_pass http://notes-api.tag16.svc.cluster.local/;
      }
      location /flaky {
        proxy_pass http://helper.tag16.svc.cluster.local/unstable?failure_rate=0.3;
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag16}
spec:
  replicas: 1
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      serviceAccountName: web
      containers:
      - name: web
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag16}
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag16}
spec:
  replicas: 1
  selector: {matchLabels: {app: last}}
  template:
    metadata: {labels: {app: last}}
    spec:
      serviceAccountName: last
      containers:
      - name: last
        image: docker.io/curlimages/curl:8.23.0
        command: [sh, -c]
        args:
        - |
          while true; do
            for p in /api/notes /api/notes /flaky; do
              code=$(curl -s -o /dev/null -w '%{http_code}' -m 5 http://web$p)
              echo "$(date +%T) $p $code"
            done
            sleep 1
          done
EOF
kubectl apply -f app.yaml
kubectl -n tag16 rollout status statefulset/postgres --timeout=180s
kubectl -n tag16 rollout status deploy/notes-api --timeout=180s
kubectl annotate namespace tag16 linkerd.io/inject=enabled \
  config.linkerd.io/default-inbound-policy=deny

The app is running, and last measures 100 percent for /api/notes. But is it in the mesh?

kubectl -n tag16 get pods -o custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd viz stat -n tag16 deploy
linkerd check --proxy -n tag16 --wait 30s 2>&1 | grep -A2 'proxies are ready'

Expected (abridged):

NAME                        PROXY
helper-79d57fbbbb-mqhpg     <none>
last-78895758f9-52xd7       <none>
notes-api-5bc98b7fb-n4s5j   <none>
postgres-0                  <none>
web-7d94ccc84d-qjrst        <none>
No traffic found.
× data plane proxies are ready
    no "linkerd-proxy" containers found in the "tag16" namespace
    see https://linkerd.io/2/checks/#l5d-data-plane-ready for hints

The webhook only kicks in when a Pod is created (Day 2). --wait 30s matters: without it, check --proxy waits up to five minutes for proxies to show up. The fix is a restart:

kubectl -n tag16 rollout restart deploy
kubectl -n tag16 rollout restart statefulset/postgres
kubectl -n tag16 rollout status statefulset/postgres --timeout=180s
sleep 45
kubectl -n tag16 get pods
kubectl -n tag16 logs deploy/last -c last --since=20s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (abridged):

NAME                         READY   STATUS       RESTARTS      AGE
helper-5c9445c87c-khfww      2/2     Running      0             53s
last-6ddfdfcc46-8cpmp        2/2     Running      0             53s
notes-api-5b6fbd4d9c-fd4n7   1/2     Init:Error   2 (38s ago)   52s
notes-api-5bc98b7fb-n4s5j    0/1     Running      0             2m3s
postgres-0                   2/2     Running      0             51s
web-6df978d4f-mhgj5          2/2     Running      0             52s
     30 /api/notes 403
     14 /flaky 403

Everything is in the mesh – and everything is broken. That is the next fault.

Save your progress? Sign up for free

Drill 2: Default deny without rules

With the restart, the default policy deny from the namespace annotation applies. What does viz show?

linkerd viz stat -n tag16 deploy
linkerd viz authz -n tag16 deploy/web

Expected:

NAME        MESHED   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
helper         1/1         -     -             -             -             -          -
last           1/1         -     -             -             -             -          -
notes-api      1/2         -     -             -             -             -          -
web            1/1         -     -             -             -             -          -
ROUTE  SERVER  AUTHORIZATION  UNAUTHORIZED  SUCCESS  RPS  LATENCY_P50  LATENCY_P95 ...

Nothing. viz reads Prometheus, and Prometheus itself is rejected on port 4191 (Day 6, Drill 6). So go straight to the proxy – via the log and diagnostics, which arrive over localhost through a port forward:

kubectl -n tag16 logs deploy/web -c linkerd-proxy --since=1m \
  | grep 'Request denied' | grep -v admin | tail -1 \
  | grep -oE 'server.name=[^ ]*|client.tls=[^ ]*'
P=$(kubectl -n tag16 get pod -l app=web -o name)
linkerd diagnostics policy -n tag16 $P 8080 -o json \
  | jq -c '{server: .labels.name, authz: .authorizations}'
linkerd diagnostics proxy-metrics -n tag16 po/postgres-0 \
  | grep '^inbound_tcp_authz_deny_total' \
  | grep -oE '(srv_name|client_id)="[^"]*"|\} [0-9]+' | paste - - -

Expected (abridged):

server.name=deny
client.tls=Some(Established
{"server":"deny","authz":null}
srv_name="deny"  client_id="notes-api.tag16.serviceaccount.identity.linkerd.cl...  } 7
srv_name="deny"  } 34

server.name=deny means: no Server matches, the default policy applies, and it contains not a single authorization. That is why Postgres also rejects migrate (Init:Error). The rules from Day 6 fit one to one, only the namespace differs:

for f in servers authz admin; do
  sed 's/namespace: notes/namespace: tag16/' ~/linkerd/tag06/$f.yaml; echo ---
done > policy.yaml
grep -c '^kind:' policy.yaml
kubectl apply -f policy.yaml
kubectl -n tag16 rollout status deploy/notes-api --timeout=180s
sleep 60
kubectl -n tag16 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n tag16 deploy

Expected (abridged, numbers vary):

14
...
deployment "notes-api" successfully rolled out
     80 /api/notes 200
      4 /api/notes 503
     29 /flaky 200
     12 /flaky 500
NAME        MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99 ...
helper         1/1    80.88%   1.1rps           1ms          14ms          19ms
last           1/1   100.00%   0.3rps           1ms          49ms          50ms
notes-api      1/1   100.00%   1.7rps           7ms          35ms          39ms
web            1/1    87.42%   2.5rps          16ms          69ms          94ms

The four 503s fall into the seconds before migrate starts again after its backoff: without a ready Pod, the proxy of web answers with 503 itself after three seconds. From now on, tag16 is healthy – the baseline for the next faults.

Warning: linkerd viz check --proxy warns here (and in notes) with “prometheus is authorized to scrape data plane pods”. The check looks for a rule for viz’s own Prometheus. With the NetworkAuthentication from Day 6, this is a false alarm – rely on up in Prometheus.

Save your progress? Sign up for free

Drill 3: The wrong ServiceAccount

A colleague redeploys last and forgets serviceAccountName:

kubectl -n tag16 set serviceaccount deploy/last default
kubectl -n tag16 rollout status deploy/last --timeout=120s
sleep 60
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz authz -n tag16 deploy/web | grep web-http
linkerd viz edges -n tag16 deploy -o wide

Expected (abridged):

     62 /api/notes 403
     30 /flaky 403
default  web-http                                    2.3rps    0.00%  0.0rps ...
default  web-http  authorizationpolicy/web-von-last  0.0rps   89.71%  1.1rps ...
SRC    DST   SRC_NS   DST_NS   CLIENT_ID       SERVER_ID   SECURED
last   web   tag16    tag16    default.tag16   web.tag16   √

The row without a rule showing 2.3 rps in UNAUTHORIZED is the hit, and edges -o wide names the identity: default.tag16. The rule, however, expects last:

kubectl -n tag16 logs deploy/web -c linkerd-proxy --since=1m \
  | grep -m1 'Request denied' | grep -oE 'server.name=[^ ]*|Name\("[^"]*'
kubectl -n tag16 get meshtlsauthentication last -o jsonpath='{.spec}{"\n"}'

Expected:

server.name=web-http
Name("default.tag16.serviceaccount.identity.linkerd.cluster.local
{"identityRefs":[{"kind":"ServiceAccount","name":"last"}]}

Tip: linkerd viz tap does not show rejected requests – they end before the point where tap listens in. For 403s, the proxy log and viz authz are the tools.

You can check which identity a Pod speaks with even without a shell in the image: an ephemeral container shares the Pod’s network and proxy. From inside helper, which is not allowed to call notes-api:

P=$(kubectl -n tag16 get pod -l app=helper -o jsonpath='{.items[0].metadata.name}')
kubectl -n tag16 debug $P -q --image=docker.io/curlimages/curl:8.23.0 -c probe -- \
  curl -s -o /dev/null -w '%{http_code}\n' -H 'X-API-Key: lab-key-123' \
  http://notes-api/notes
sleep 5
kubectl -n tag16 logs $P -c probe
kubectl -n tag16 logs deploy/notes-api -c linkerd-proxy --since=1m \
  | grep 'Request denied' | tail -1 | grep -oE 'Name\("[^"]*'

Expected:

403
Name("helper.tag16.serviceaccount.identity.linkerd.cluster.local

The debug container has the Pod’s identity – handy for testing a rule from the point of view of exactly this client. Now repair last:

kubectl -n tag16 set serviceaccount deploy/last last
kubectl -n tag16 rollout status deploy/last --timeout=120s
sleep 45
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (numbers vary):

     54 /api/notes 200
     21 /flaky 200
      6 /flaky 500

You get the same picture when the MeshTLSAuthentication contains a typo or the ServiceAccount comes from a different namespace (identityRefs with namespace).

Save your progress? Sign up for free

Drill 4: Ten seconds of silence

A service where the server speaks first – like SMTP or MySQL. It announces itself with a greeting line on port 2525, which is not on Linkerd’s opaque list:

kubectl -n tag16 create serviceaccount banner
cat > banner.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: banner, namespace: tag16}
spec:
  replicas: 1
  selector: {matchLabels: {app: banner}}
  template:
    metadata: {labels: {app: banner}}
    spec:
      serviceAccountName: banner
      containers:
      - name: banner
        image: docker.io/library/busybox:1.37
        command: [nc, -lk, -p, "2525", -e, echo, "220 banner bereit"]
        ports: [{containerPort: 2525, name: smtp}]
---
apiVersion: v1
kind: Service
metadata: {name: banner, namespace: tag16}
spec:
  selector: {app: banner}
  ports: [{port: 2525, targetPort: smtp, name: smtp}]
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: banner, namespace: tag16}
spec:
  podSelector: {matchLabels: {app: banner}}
  port: smtp
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: banner-von-last, namespace: tag16}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: banner}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: last}
EOF
kubectl apply -f banner.yaml
kubectl -n tag16 rollout status deploy/banner --timeout=120s
kubectl -n tag16 exec deploy/last -c last -- sh -c 's=$(date +%s); (sleep 15) \
  | nc banner 2525 | while read l; do echo "$(( $(date +%s) - s )) s: $l"; done'

Expected:

10 s: 220 banner bereit

Exactly ten seconds. The client proxy waits for the client’s first bytes to detect the protocol. The client waits for the server. Both wait until the proxy gives up:

kubectl -n tag16 logs deploy/last -c linkerd-proxy --since=1m | grep 'after timeout'
linkerd diagnostics policy -n tag16 svc/banner 2525 -o json \
  | jq -c '.protocol.Kind | keys, .Detect.timeout'

Expected (abridged):

INFO ThreadId(01) outbound:proxy{addr=10.99.111.91:2525}: linkerd_app_outbound::pro...
  ...tocol: Continuing after timeout: 10s
["Detect"]
{"seconds":10}

diagnostics policy with a Service shows the outbound view: protocol Detect with a ten-second deadline. So: opaque port, on the Service:

kubectl -n tag16 annotate service banner config.linkerd.io/opaque-ports=2525
sleep 5
linkerd diagnostics policy -n tag16 svc/banner 2525 -o json \
  | jq -c '.protocol.Kind | keys'
kubectl -n tag16 exec deploy/last -c last -- sh -c 's=$(date +%s); (sleep 3) \
  | nc banner 2525 | while read l; do echo "$(( $(date +%s) - s )) s: $l"; done'
linkerd check --proxy -n tag16 --wait 30s 2>&1 | grep -A2 opaque

Expected (abridged):

["Opaque"]
0 s: 220 banner bereit
‼ opaque ports are properly annotated
    * service banner expects target port 2525 to be opaque; add it to pod
      banner-84d6cdb7b8-pl2kb config.linkerd.io/opaque-ports annotation

For clients in the mesh, the Service is enough. check also wants the annotation on the Pod – for direct Pod calls and headless Services:

kubectl -n tag16 patch deploy banner -p '{"spec":{"template":{"metadata":
  {"annotations":{"config.linkerd.io/opaque-ports":"2525"}}}}}'
kubectl -n tag16 rollout status deploy/banner --timeout=120s
linkerd check --proxy -n tag16 --wait 30s 2>&1 | grep -E 'opaque|Status check'

Expected:

√ opaque ports are properly annotated
Status check results are √

Warning: The annotation config.linkerd.io/opaque-ports on the workload replaces the default list (25, 587, 3306, …) for this Pod instead of extending it. As an alternative on the Service, you can use appProtocol: linkerd.io/opaque on the port.

Save your progress? Sign up for free

Drill 5: Bypassing the proxy

Someone wanted to make the database connection “faster” and takes port 5432 out of the proxy:

kubectl -n tag16 patch deploy notes-api -p '{"spec":{"template":{"metadata":
  {"annotations":{"config.linkerd.io/skip-outbound-ports":"5432"}}}}}'
sleep 50
kubectl -n tag16 get pods -l app=notes-api

Expected:

NAME                         READY   STATUS       RESTARTS      AGE
notes-api-5b6fbd4d9c-fd4n7   2/2     Running      0             9m37s
notes-api-794d4966dd-bzmxp   1/2     Init:Error   2 (42s ago)   50s

The old Pod keeps running, users notice nothing – the rollout is stuck. Who is rejecting?

P=$(kubectl -n tag16 get pod -l app=notes-api \
  --sort-by=.metadata.creationTimestamp -o name | tail -1)
kubectl -n tag16 logs $P -c migrate --tail=1
linkerd diagnostics proxy-metrics -n tag16 po/postgres-0 \
  | grep '^inbound_tcp_authz_deny_total' \
  | grep -oE '(srv_name|no_tls_reason)="[^"]*"|\} [0-9]+' | paste - - -
kubectl -n tag16 get $P -o jsonpath='{.spec.initContainers[0].args}' \
  | grep -o '"--outbound-ports-to-ignore","[^"]*"'

Expected (abridged):

time=2026-11-19T20:05:56.868Z level=ERROR msg="database unreachable" source=env
err="failed to connect to `user=notes database=notes`: 10.100.223.86:5432 (postgres):
failed to receive message: unexpected EOF"
srv_name="postgres"  no_tls_reason="port_skipped"  } 16
"--outbound-ports-to-ignore","5432"

port_skipped means: plaintext on an opaque port. linkerd-init routed 5432 past the iptables rules, so notes-api talks without a proxy and without an identity. Revert it:

kubectl -n tag16 patch deploy notes-api --type=json -p '[{"op":"remove",
  "path":"/spec/template/metadata/annotations/config.linkerd.io~1skip-outbound-ports"}]'
kubectl -n tag16 rollout status deploy/notes-api --timeout=180s
kubectl -n tag16 get pods -l app=notes-api

Expected (abridged):

deployment "notes-api" successfully rolled out
NAME                         READY   STATUS        RESTARTS      AGE
notes-api-5b6fbd4d9c-fd4n7   2/2     Running       0             9m48s
notes-api-794d4966dd-bzmxp   1/2     Terminating   3 (38s ago)   61s

An init container without a native sidecar (Day 6, Drill 7) produces exactly the same picture – Init:Error, port_skipped. The telltale signs there: the annotation config.beta.linkerd.io/proxy-enable-native-sidecar: "false" and linkerd-init after your init container in spec.initContainers.

Tip: The arguments of linkerd-init also contain the default --inbound-ports-to-ignore 4190,4191,4567,4568. So Linkerd already accepts nothing inbound on 4567/4568 (Galera) on its own.

Save your progress? Sign up for free

Drill 6: The route that does nothing

Retries for helper as on Day 7 – with a small mistake in the parentRef:

cat > route-retry.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: helper-retry
  namespace: tag16
  annotations:
    retry.linkerd.io/http: 5xx
    retry.linkerd.io/limit: "2"
spec:
  parentRefs:
  - {group: core, kind: Service, name: helper, port: 8080}
  rules:
  - matches:
    - path: {type: PathPrefix, value: /}
      method: GET
EOF
kubectl apply -f route-retry.yaml
sleep 65
kubectl -n tag16 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag16 get httproute helper-retry \
  -o jsonpath='{.status.parents[0].conditions[*].reason}{"\n"}'

Expected (numbers vary):

     92 /api/notes 200
     33 /flaky 200
     13 /flaky 500
Accepted ResolvedRefs

Still 70 percent, even though the route is Accepted. The policy controller does not check whether the Service even has port 8080. What the proxy actually receives:

linkerd diagnostics policy -n tag16 svc/helper 80 -o json \
  | jq -c '.protocol.Kind.Detect.http1.routes[].metadata.Kind'
linkerd diagnostics policy -n tag16 svc/helper 8080 -o json \
  | jq -c '.protocol.Kind.Detect.http1.routes[].metadata.Kind'
linkerd diagnostics proxy-metrics -n tag16 deploy/web \
  | grep -c '^outbound_http_route_retry_requests_total'

Expected:

{"Default":"http"}
{"Resource":{"group":"gateway.networking.k8s.io","kind":"HTTPRoute","name":"helper-r...
0

parentRef.port is the Service port (80), not the container port (8080). On port 80, the proxy sees only the default route, zero retries.

sed -i 's/port: 8080}/port: 80}/' route-retry.yaml
kubectl apply -f route-retry.yaml
sleep 65
kubectl -n tag16 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd diagnostics proxy-metrics -n tag16 deploy/web \
  | grep '^outbound_http_route_retry_requests_total' \
  | grep -oE 'route_name="[^"]*"|[0-9]+$' | paste - -

Expected (numbers vary):

     84 /api/notes 200
     40 /flaky 200
      2 /flaky 500
route_name="helper-retry"  14

Other candidates when a route does nothing: a forgotten group (if omitted, gateway.networking.k8s.io applies), a route in a different namespace than the Service, a ServiceProfile for the same Service (it wins, Day 7), or a client without a proxy – only the client proxy performs retries. A retry budget as in the old ServiceProfile does not exist on HTTPRoutes, only limit.

Save your progress? Sign up for free

Drill 7: A renamed port

A developer renames the container port in helper from http to web and updates the probe accordingly:

kubectl -n tag16 patch deploy helper --type=json -p '[
  {"op":"replace","path":"/spec/template/spec/containers/0/ports/0/name",
   "value":"web"},
  {"op":"replace",
   "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port",
   "value":"web"}]'
kubectl -n tag16 rollout status deploy/helper --timeout=120s
sleep 50
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80
kubectl -n tag16 get endpointslices -l kubernetes.io/service-name=helper

Expected:

     58 /api/notes 200
     29 /flaky 503
No endpoints found.
NAME           ADDRESSTYPE   PORTS     ENDPOINTS    AGE
helper-jcdjm   IPv4          <unset>   10.0.1.106   60s

The Pod is ready, but the Service looks for targetPort: http – which no longer exists. diagnostics endpoints shows what linkerd-destination hands out to the proxies: nothing. For more detail in the proxy of web, raise the log level. The annotation only takes effect after a restart and produces a lot of text – use it only briefly:

kubectl -n tag16 patch deploy web -p '{"spec":{"template":{"metadata":
  {"annotations":{"config.linkerd.io/proxy-log-level":"warn,linkerd=debug"}}}}}'
kubectl -n tag16 rollout status deploy/web --timeout=120s
sleep 30
kubectl -n tag16 logs deploy/web -c linkerd-proxy --since=30s \
  | grep -oE 'name=helper port=80\}:pool: .*' | cut -c1-80 | sort | uniq -c \
  | sort -rn | head -4

Expected (abridged):

     21 name=helper port=80}:pool: linkerd_pool_p2c: No ready endpoints
      7 name=helper port=80}:pool: linkerd_proxy_api_resolve::resolve: Resolving ds
      6 name=helper port=80}:pool: linkerd_proxy_resolve::recover: recovering err=c
      1 name=helper port=80}:pool: linkerd_proxy_balance_queue::worker: Unavailable

“No ready endpoints”, then fail-fast after three seconds: hence the 503s. The obvious repair is to update the Service to match:

kubectl -n tag16 patch service helper --type=json \
  -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":"web"}]'
sleep 20
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80
kubectl -n tag16 logs deploy/last -c last --since=12s | awk '{print $2, $3}' \
  | sort | uniq -c
P=$(kubectl -n tag16 get pod -l app=helper -o name)
linkerd diagnostics policy -n tag16 $P 8080 -o json | jq -c '{server: .labels.name}'

Expected:

NAMESPACE   IP           PORT   POD                      SERVICE
tag16       10.0.1.106   8080   helper-9d8c45bb8-gsbm5   helper.tag16
     18 /api/notes 200
      9 /flaky 403
{"server":"deny"}

Endpoints are there, now 403: the Server helper-http also looks for port: http. The port falls under the default policy. Port names are a contract between Pod, Service, and Server. The right fix is to rename the port back – and revert the log level at the same time:

kubectl -n tag16 patch deploy helper --type=json -p '[
  {"op":"replace","path":"/spec/template/spec/containers/0/ports/0/name",
   "value":"http"},
  {"op":"replace",
   "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port",
   "value":"http"}]'
kubectl -n tag16 patch service helper --type=json \
  -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":"http"}]'
kubectl -n tag16 patch deploy web --type=json -p '[{"op":"remove",
  "path":"/spec/template/metadata/annotations/config.linkerd.io~1proxy-log-level"}]'
kubectl -n tag16 rollout status deploy/helper --timeout=120s
kubectl -n tag16 rollout status deploy/web --timeout=120s
sleep 50
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c
P=$(kubectl -n tag16 get pod -l app=helper -o name)
linkerd diagnostics policy -n tag16 $P 8080 -o json | jq -c '{server: .labels.name}'

Expected (numbers vary):

     59 /api/notes 200
     28 /flaky 200
      1 /flaky 500
{"server":"helper-http"}

Save your progress? Sign up for free

Drill 8: Proxy OOMKilled

You set resources for the proxy via annotation. A limit that is too tight goes unnoticed while idle – the proxy here needs only about 3 MiB:

kubectl top pod -n tag16 --containers | grep -E 'POD|helper.*proxy'
kubectl -n tag16 patch deploy helper -p '{"spec":{"template":{"metadata":
  {"annotations":{"config.linkerd.io/proxy-memory-limit":"6Mi"}}}}}'
kubectl -n tag16 rollout status deploy/helper --timeout=120s
kubectl -n tag16 get pods -l app=helper

Expected (abridged):

POD                          NAME            CPU(cores)   MEMORY(bytes)
helper-5c9445c87c-khfww      linkerd-proxy   4m           3Mi
NAME                      READY   STATUS    RESTARTS   AGE
helper-665b47786f-lt26b   2/2     Running   0          45s

It runs. Then load arrives: 200 concurrent, slow requests from web (go-httpbin /drip trickles the response over five seconds):

kubectl -n tag16 exec deploy/web -c web -- sh -c 'seq 400 | xargs -P 200 -I{} \
  curl -s -o /dev/null -w "%{http_code}\n" -m 20 \
  "http://helper/drip?duration=5&numbytes=5&delay=0"' | sort | uniq -c
kubectl -n tag16 get pods -l app=helper
kubectl -n tag16 get pod -l app=helper -o custom-columns=NAME:.metadata.name,\
PROXY-RESTARTS:.status.initContainerStatuses[1].restartCount,\
GRUND:.status.initContainerStatuses[1].lastState.terminated.reason
kubectl -n tag16 get pod -l app=helper \
  -o jsonpath='{.items[0].spec.initContainers[1].resources}{"\n"}'

Expected (abridged, numbers vary):

command terminated with exit code 123
    199 200
    201 503
NAME                      READY   STATUS    RESTARTS      AGE
helper-665b47786f-5hwgq   2/2     Running   1 (11s ago)   30s
NAME                      PROXY-RESTARTS   GRUND
helper-665b47786f-5hwgq   1                OOMKilled
{"limits":{"memory":"6Mi"},"requests":{"memory":"6Mi"}}

Exit code 123 is reported only by xargs, because individual curl calls failed. The proxy is a long-running init container, which is why the reason appears under initContainerStatuses. If you catch the Pod in backoff, its status reads Init:CrashLoopBackOff. While the proxy restarts, helper is gone: 503 from the proxy in web. Afterwards the Pod comes back up – until the next load spike. Without a request annotation, Kubernetes sets the request equal to the limit. Set limits after measuring under load, with headroom:

kubectl -n tag16 patch deploy helper -p '{"spec":{"template":{"metadata":
  {"annotations":{"config.linkerd.io/proxy-memory-limit":"128Mi",
  "config.linkerd.io/proxy-memory-request":"20Mi"}}}}}'
kubectl -n tag16 rollout status deploy/helper --timeout=120s
kubectl -n tag16 exec deploy/web -c web -- sh -c 'seq 400 | xargs -P 200 -I{} \
  curl -s -o /dev/null -w "%{http_code}\n" -m 20 \
  "http://helper/drip?duration=5&numbytes=5&delay=0"' | sort | uniq -c
kubectl -n tag16 get pods -l app=helper
kubectl top pod -n tag16 --containers | grep helper

Expected (abridged):

    400 200
NAME                      READY   STATUS    RESTARTS   AGE
helper-5bf6f997c8-zdqlt   2/2     Running   0          26s
helper-5bf6f997c8-zdqlt      helper          58m          9Mi
helper-5bf6f997c8-zdqlt      linkerd-proxy   63m          7Mi

The proxy’s memory grows with concurrent connections and requests, not with the number of Pods in the cluster.

Tip: If the proxy-memory-request is greater than the limit, creating the Pod already fails (FailedCreate, “must be less than or equal to memory limit” – also for linkerd-init). rollout status then just waits silently; the cause is in kubectl get events.

Warning: Better not reproduce certificate and clock errors – they hit the entire mesh. If the issuer expires, new proxies no longer get an identity (linkerd check reports it under linkerd-identity, the proxy log shows errors fetching the certificate, Pods do not become Ready). If a node’s clock is wrong, certificates there are “not yet” or “no longer” valid: mTLS handshakes fail only from and to Pods on that node. Early warning: control_identity_cert_refreshes_total{result="error"} (Day 3), expiry dates of issuer and anchor (Day 5), and NTP on all nodes.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show which Pods in tag16 have a proxy and which version. Target time: 20 s
  2. Show which Server and which rules apply to port 5432 in postgres-0. Target time: 40 s
  3. Show which identities notes-api has rejected so far and how often. Target time: 60 s
  4. Show when the issuer and the proxy certificate of web expire. Target time: 40 s

Solution

kubectl -n tag16 get pods -o custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd diagnostics policy -n tag16 po/postgres-0 5432 -o json \
  | jq -c '{server: .labels.name, authz: [.authorizations[].labels.name]}'
linkerd diagnostics proxy-metrics -n tag16 deploy/notes-api \
  | grep '^inbound_http_authz_deny_total' \
  | sed -E 's/.*client_id="([^"]*)".*\} /\1 /; s/^inbound.*\} /ohne-identitaet /'
kubectl -n linkerd get certificate linkerd-identity-issuer \
  -o jsonpath='{.status.notAfter}{"\n"}'
linkerd identity -n tag16 -l app=web | grep 'Not After'

Expected (abridged):

NAME                         PROXY
banner-f67576cf9-6kptm       edge-26.11.2
...
{"server":"postgres","authz":["postgres-von-notes-api"]}
helper.tag16.serviceaccount.identity.linkerd.cluster.local 1
ohne-identitaet 6
2026-11-21T16:02:00Z
            Not After : Nov 20 20:19:15 2026 UTC

ohne-identitaet (without identity) are the Prometheus scrapes on 4191 from Drill 2, before proxy-admin was in place. The issuer is valid for 48 hours and is renewed by cert-manager (Day 5).

Save your progress? Sign up for free

Checkpoint

kubectl -n tag16 get pods

Expected: all Pods 2/2 Running.

kubectl -n tag16 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected: /api/notes only 200, /flaky almost only 200 (retries from Drill 6).

linkerd check --proxy -n tag16 2>&1 | tail -1

Expected:

Status check results are √
kubectl -n tag16 get pods -o custom-columns=NAME:.metadata.name,\
PROXY-RESTARTS:.status.initContainerStatuses[1].restartCount

Expected: a 0 after every Pod – no proxy restart since the last rollout.

Challenge

Roll out a “release” that changes two things at once, and then repair everything, using only Linkerd tools and without looking at the commands below again:

kubectl -n tag16 create serviceaccount web-v2
kubectl -n tag16 set serviceaccount deploy/web web-v2
kubectl -n tag16 patch service helper -p '{"spec":{"selector":{"app":"helper-v2"}}}'

web should keep web-v2.

Hint: Two paths, two different status codes – two causes.

Solution

sleep 50
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag16 logs deploy/notes-api -c linkerd-proxy --since=1m \
  | grep -m1 'Request denied' | grep -oE 'server.name=[^ ]*|Name\("[^"]*'
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80
kubectl -n tag16 get service helper -o jsonpath='{.spec.selector}{"\n"}'

Expected:

     59 /api/notes 403
     29 /flaky 503
server.name=notes-api-http
Name("web-v2.tag16.serviceaccount.identity.linkerd.cluster.local
No endpoints found.
{"app":"helper-v2"}

403 on /api/notes: notes-api sees the new identity web-v2, but the rule only knows web. 503 on /flaky: the Service finds no Pod. The MeshTLSAuthentication web is used by two rules (notes-api-von-web, helper-von-web) – one change is enough for both:

kubectl -n tag16 patch meshtlsauthentication web --type=merge -p '{"spec":
  {"identityRefs":[{"kind":"ServiceAccount","name":"web"},
  {"kind":"ServiceAccount","name":"web-v2"}]}}'
kubectl -n tag16 patch service helper -p '{"spec":{"selector":{"app":"helper"}}}'
sleep 50
kubectl -n tag16 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80

Expected (numbers vary):

     48 /api/notes 200
     22 /flaky 200
      1 /flaky 500
NAMESPACE   IP           PORT   POD                       SERVICE
tag16       10.0.1.175   8080   helper-5bf6f997c8-l4fkh   helper.tag16

Had you looked only for the 403, /flaky would have stayed at 503 – and vice versa. With multiple symptoms, trace each one individually down to its cause.

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag16
kubectl get endpointslices -A | grep cilium-gateway | grep tag16

Expected: no output. Along with the namespace, the PVC and the EndpointSlice cilium-gateway-helper, which Cilium created for the route from Drill 6, disappear too.

Interview Note

  • How do you approach a mesh outage? From the outside in: measure the symptom, linkerd check --proxy, viz metrics, then diagnostics policy/endpoints as the proxy’s view, and finally logs with a raised log level.
  • 403 or 503 – where do you look? 403 is policy on the server side (Server, identity, route). 503 from the proxy usually means: no ready endpoints, fail-fast in the client proxy.
  • Why does a connection hang for exactly ten seconds? Protocol detection with a protocol in which the server speaks first – set an opaque port.

Enterprise Note

  • Runbooks per symptom like the table above, with the exact commands. During an outage, nobody reads the docs.
  • Alerts on inbound_http_authz_deny_total, inbound_tcp_authz_deny_total, proxy restarts (kube_pod_init_container_status_restarts_total), and retry rates.
  • Never keep debug logging permanently: it costs CPU, memory, and log volume. Set the annotation via a change and remove it again.
  • Check port names, ServiceAccounts, and opaque ports with an admission policy (Kyverno, Gatekeeper) – most of today’s faults can be caught before deployment.
  • Restrict kubectl debug via RBAC: a debug container inherits the Pod’s identity and thus all of its permissions in the mesh.