Skip to content
Hands-on IT IT Drills

Day 7: Retries and Timeouts

Goal: /flaky succeeds almost 100 percent of the time with retries, /slow aborts after one second – and you know what retries cost and when they are dangerous. · Duration: approx. 60 min · Prerequisite: Day 6

Briefing

  • Retries and timeouts are handled by the client proxy (outbound). The app does not notice anything.
  • You configure them through a Gateway API HTTPRoute with the Service as its parentRef and annotations: retry.linkerd.io/http, retry.linkerd.io/limit, retry.linkerd.io/timeout, timeout.linkerd.io/request (or response, idle).
  • Once a route is attached to a Service, only what a route matches still applies to it.
  • A retry is an additional request: more load, and with POST possibly a duplicate order.
 last ──> web ──(outbound proxy: route, retry, timeout)──> helper (2 Pods)
                    /unstable  retry on 5xx, max. 2 retries
                    /delay     timeout 1 s

Drill 1: A copy for experiments

notes stays untouched. In tag07, a small copy of web, helper and last runs with its own ServiceAccounts. Here the load generator calls /flaky twice and /slow once.

mkdir -p ~/linkerd/tag07 && cd ~/linkerd/tag07
kubectl create namespace tag07
kubectl annotate namespace tag07 linkerd.io/inject=enabled
for a in web helper last; do kubectl -n tag07 create serviceaccount $a; done
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag07}
data:
  default.conf: |
    server {
      listen 8080;
      location /flaky {
        proxy_pass http://helper.tag07.svc.cluster.local/unstable?failure_rate=0.3;
      }
      location /slow {
        proxy_pass http://helper.tag07.svc.cluster.local/delay/2;
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag07}
spec:
  replicas: 1
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      serviceAccountName: web
      containers:
      - name: web
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag07}
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: tag07}
spec:
  replicas: 2
  selector: {matchLabels: {app: helper}}
  template:
    metadata: {labels: {app: helper}}
    spec:
      serviceAccountName: helper
      containers:
      - name: helper
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: tag07}
spec:
  selector: {app: helper}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag07}
spec:
  replicas: 1
  selector: {matchLabels: {app: last}}
  template:
    metadata: {labels: {app: last}}
    spec:
      serviceAccountName: last
      containers:
      - name: last
        image: docker.io/curlimages/curl:8.23.0
        command: [sh, -c]
        args:
        - |
          while true; do
            for p in /flaky /flaky /slow; do
              code=$(curl -s -o /dev/null -w "%{http_code}" -m 5 http://web$p)
              echo "$(date +%T) $p $code"
            done
            sleep 1
          done
EOF
kubectl apply -f app.yaml
kubectl -n tag07 rollout status deploy/helper --timeout=180s
kubectl -n tag07 get pods

Expected (abridged):

NAME                      READY   STATUS    RESTARTS   AGE
helper-79d57fbbbb-cxfnr   2/2     Running   0          13s
helper-79d57fbbbb-wknd2   2/2     Running   0          14s
last-84d85f57b9-rbfjf     2/2     Running   0          14s
web-7d94ccc84d-bgq62      2/2     Running   0          14s

For measuring per route there is linkerd viz stat-outbound. It queries Prometheus from your machine – with the Prometheus from monitoring, that means through a port-forward that you leave open for today:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
PROM=--prometheusURL=http://localhost:9090
sleep 60
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat-outbound -n tag07 deploy/web $PROM

Expected (abridged, numbers vary):

     20 /flaky 200
     14 /flaky 500
     17 /slow 200
NAME  SERVICE    ROUTE      TYPE  BACKEND    SUCCESS   RPS  ...  TIMEOUTS  RETRIES
web   helper:80  [default]                    76.00%  0.83  ...     0.00%    0.00%
                 └─────────────►  helper:80   76.00%  0.83  ...     0.00%

Without $PROM, the CLI tries to resolve the Service name from the viz configuration directly and fails with lookup kps-kube-prometheus-stack-prometheus... server misbehaving. [default] is the route Linkerd assumes when there is no HTTPRoute of your own.

Save your progress? Sign up for free

Drill 2: Retries for /unstable

cat > route-retry.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: helper-unstable
  namespace: tag07
  annotations:
    retry.linkerd.io/http: 5xx
    retry.linkerd.io/limit: "2"
spec:
  parentRefs:
  - {group: core, kind: Service, name: helper, port: 80}
  rules:
  - matches:
    - path: {type: PathPrefix, value: /unstable}
EOF
kubectl apply -f route-retry.yaml
kubectl -n tag07 get httproute helper-unstable -o yaml | grep -E 'controllerName|reason'
sleep 70
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat-outbound -n tag07 deploy/web $PROM

Expected (abridged, numbers vary):

      reason: Accepted
      reason: ResolvedRefs
    controllerName: linkerd.io/policy-controller
     83 /flaky 200
      1 /flaky 500
     42 /slow 404
ROUTE            TYPE       BACKEND    SUCCESS   RPS  ...  TIMEOUTS  RETRIES
helper-unstable  HTTPRoute             100.00%  1.35  ...     0.00%   27.03%
└────────────────────────►  helper:80   72.32%  1.87  ...     0.00%

Warning: Cilium reads HTTPRoutes too, and for a route with a Service as its parentRef it creates an EndpointSlice cilium-gateway-<service> with the placeholder address 192.192.192.192:9999. It belongs to no real Service and does not interfere with Linkerd, but it stays behind after you delete the route. Only Linkerd sets the route’s status. To show it: kubectl get endpointslices -A | grep cilium-gateway.

/flaky jumps from around 70 to almost 100 percent: three attempts fail only with a probability of 0.3 × 0.3 × 0.3 ≈ 3 percent. The route’s line shows what web experiences, the line below it what helper really delivers – still 72 percent. And /slow?

Save your progress? Sign up for free

Drill 3: The trap with the remaining paths

/slow suddenly returns 404, faster than before. As soon as an HTTPRoute is attached to the Service, it replaces the default route. The client proxy itself answers paths that no rule matches with 404. So you need a route for the rest:

cat > route-rest.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: helper-rest, namespace: tag07}
spec:
  parentRefs:
  - {group: core, kind: Service, name: helper, port: 80}
  rules:
  - matches:
    - path: {type: PathPrefix, value: /}
EOF
kubectl apply -f route-rest.yaml
sleep 70
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat-outbound -n tag07 deploy/web $PROM | grep -v '└'

Expected (abridged):

     36 /flaky 200
     17 /slow 200
ROUTE            TYPE       BACKEND    SUCCESS   RPS  ...  TIMEOUTS  RETRIES
helper-unstable  HTTPRoute             100.00%  0.55  ...     0.00%   26.67%
helper-rest      HTTPRoute             100.00%  0.30  ...     0.00%    0.00%

The Gateway API picks the most specific rule: /unstable wins over /.

Warning: In notes, the same mistake would have taken down every path except that one. Whoever attaches the first route to a Service always adds the one for / right along with it.

Save your progress? Sign up for free

Drill 4: The price

You already saw it in Drill 2: web sends 1.35 requests per second, helper receives 1.87. Retries are load. How bad it gets when helper fails completely is shown by a call directly from last – its proxy knows the same route:

kubectl -n tag07 exec deploy/last -c last -- sh -c 'for i in $(seq 20); do
  curl -s -o /dev/null -w "%{http_code}\n" "http://helper/unstable?failure_rate=1"
  done' | sort | uniq -c
linkerd diagnostics proxy-metrics -n tag07 deploy/last \
  | grep -E '^outbound_http_route_retry_(requests|limit_exceeded)_total' \
  | sed -E 's/\{.*\}//'

Expected:

     20 500
outbound_http_route_retry_limit_exceeded_total 20
outbound_http_route_retry_requests_total 40

last never called helper directly before, so the counters belong to this test only. 20 requests, 40 retries: helper sees three times the load – exactly when it is already down. In chains, this multiplies: three services with two retries each turn one request into up to 27.

In the past, a retry budget in the ServiceProfile limited this (retryRatio: 0.2 meant: at most 20 percent additional load). The annotations on HTTPRoutes know no budget, only limit. Keep it small and rely on circuit breaking (Day 8) against persistent failures. You cannot mix the two: if a ServiceProfile exists for a Service, the proxy ignores the annotations.

Save your progress? Sign up for free

Drill 5: Timeouts

/slow hangs for two seconds. Your goal: give up after one second.

cat > route-delay.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: helper-delay
  namespace: tag07
  annotations:
    timeout.linkerd.io/request: 1s
spec:
  parentRefs:
  - {group: core, kind: Service, name: helper, port: 80}
  rules:
  - matches:
    - path: {type: PathPrefix, value: /delay}
EOF
kubectl apply -f route-delay.yaml
sleep 5
kubectl -n tag07 exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://web/slow
kubectl -n tag07 logs deploy/web -c linkerd-proxy --since=30s \
  | grep -o 'route HTTPRoute[^:]*: [^,]*request timeout: 1s' | tail -1

Expected (abridged):

504 1.019408s
route HTTPRoute.tag07.helper-delay: backend Service.tag07.helper:80: ...

The proxy of web aborts after one second and answers 504 Gateway Timeout, which nginx passes through. After one minute:

sleep 60
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat-outbound -n tag07 deploy/web $PROM | grep helper-delay

Expected (abridged):

     49 /flaky 200
      1 /flaky 500
     24 /slow 504
helper-delay  HTTPRoute    0.00%  0.42  ...   100.00%    0.00%

The TIMEOUTS column shows 100 percent. The latency columns in stat-outbound say little for timeouts – they are estimated from coarse histogram buckets. A timeout is not retried. If you want that, set retry.linkerd.io/timeout (time per attempt) together with retry.linkerd.io/http.

Save your progress? Sign up for free

Drill 6: POST is not idempotent

The route helper-unstable matches every method. What happens to an order?

timeout 15 linkerd viz tap -n tag07 deploy/helper --method POST > tap-post.txt &
sleep 5
kubectl -n tag07 exec deploy/last -c last -- curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST -d 'bestellung=1' 'http://helper/unstable?failure_rate=1'
sleep 12
grep -c '^req' tap-post.txt

Expected:

500
3

One POST from the client, three at helper. Had the first attempt stored the order and only then answered with 500, it would now exist three times. Linkerd does not check the method – it only never retries bodies over 64 KiB. So only GET:

sed -i 's|    - path: {type: PathPrefix, value: /unstable}|&\n      method: GET|' \
  route-retry.yaml
tail -3 route-retry.yaml
kubectl apply -f route-retry.yaml
sleep 5
timeout 15 linkerd viz tap -n tag07 deploy/helper --method POST > tap-post.txt &
sleep 5
kubectl -n tag07 exec deploy/last -c last -- curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST -d 'bestellung=1' 'http://helper/unstable?failure_rate=1'
sleep 12
grep -c '^req' tap-post.txt

Expected:

  - matches:
    - path: {type: PathPrefix, value: /unstable}
      method: GET
httproute.gateway.networking.k8s.io/helper-unstable configured
500
1

The POST now falls through to helper-rest without retries. GET, HEAD, PUT and DELETE are idempotent according to HTTP – whether your app implements them that way too is something you check before you turn on retries.

Save your progress? Sign up for free

Drill 7: Retries in Prometheus

For dashboards and alerts, you count directly:

Q='sum by (route_name) (rate(outbound_http_route_retry_requests_total{
 namespace="tag07", deployment="web"}[2m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
  | jq -r '.data.result[] | .metric.route_name + " " + .value[1]'
Q='sum by (route_name, http_status) (rate(outbound_http_route_request_statuses_total{
 namespace="tag07", deployment="web"}[2m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
  | jq -r '.data.result[] | .metric.route_name + " " + .metric.http_status
  + " " + .value[1]' | grep -v ' 0$'

Expected (abridged, numbers vary):

helper-unstable 0.23636363636363636
helper-unstable 200 0.7999999999999999
helper-delay  0.3818181818181818
helper-delay 200 0.027166000000000003

Aborted requests have no http_status – the gap at helper-delay is the timeouts. A rising ratio of retries to requests is an early alarm: the service behind it is getting worse, even if users do not notice anything yet.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all HTTPRoutes in tag07 with their Linkerd annotations. Target time: 30 s
  2. Raise the retry limit of helper-unstable to 3 without changing the file. Target time: 30 s
  3. Show how many retries the proxy of web has sent so far. Target time: 30 s
  4. Show whether the policy controller has accepted all routes. Target time: 45 s

Solution

kubectl -n tag07 describe httproute | grep -E '^Name:|linkerd.io/'
kubectl -n tag07 annotate httproute helper-unstable retry.linkerd.io/limit=3 --overwrite
linkerd diagnostics proxy-metrics -n tag07 deploy/web \
  | grep '^outbound_http_route_retry_requests_total' | grep -oE '[0-9]+$'
kubectl -n tag07 get httproute \
  -o custom-columns=NAME:.metadata.name,STATUS:.status.parents[0].conditions[0].reason

Expected (abridged):

Name:         helper-delay
Annotations:  timeout.linkerd.io/request: 1s
...
Name:         helper-unstable
Annotations:  retry.linkerd.io/http: 5xx
              retry.linkerd.io/limit: 2
httproute.gateway.networking.k8s.io/helper-unstable annotated
145
NAME              STATUS
helper-delay      Accepted
helper-rest       Accepted
helper-unstable   Accepted

Save your progress? Sign up for free

Checkpoint

kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected: /flaky almost only 200, /slow only 504.

     48 /flaky 200
     25 /slow 504
linkerd viz stat-outbound -n tag07 deploy/web $PROM | grep -c HTTPRoute

Expected: 2 – helper-unstable and helper-delay. Nothing from the load generator goes through helper-rest anymore, since /delay has its own route.

Challenge

Change the timeout for /delay to three seconds – this time using the Gateway API field instead of the annotation – and find out what applies when both are set.

Hint: The field is called timeouts.request and sits in the rule (rules[]).

Solution

kubectl -n tag07 annotate httproute helper-delay timeout.linkerd.io/request-
kubectl -n tag07 patch httproute helper-delay --type=json \
  -p '[{"op":"add","path":"/spec/rules/0/timeouts","value":{"request":"3s"}}]'
sleep 5
kubectl -n tag07 exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://web/slow
kubectl -n tag07 annotate httproute helper-delay timeout.linkerd.io/request=1s
sleep 5
kubectl -n tag07 exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://web/slow

Expected:

httproute.gateway.networking.k8s.io/helper-delay annotated
httproute.gateway.networking.k8s.io/helper-delay patched
200 2.026916s
httproute.gateway.networking.k8s.io/helper-delay annotated
504 1.039460s

With 3 s, /slow gets through again. If both the field and the annotation are set, the annotation wins. For response and idle timeouts, only the annotations exist.

Save your progress? Sign up for free

Cleanup

kill %1
kubectl delete namespace tag07
kubectl get endpointslices -A | grep cilium-gateway

Expected: no line from tag07 anymore – Cilium’s EndpointSlice disappears along with the namespace. If you only delete a route, you remove it yourself: kubectl -n <ns> delete endpointslice cilium-gateway-<service>.

Interview Note

  • Where do retries run in Linkerd? In the client proxy, configured via HTTPRoute or Service annotation (retry.linkerd.io/*). The app does not notice anything.
  • Why not retries everywhere? They multiply the load exactly when things fail and are dangerous with non-idempotent methods such as POST.
  • Timeout or retry first? Both: the timeout limits the user’s waiting time, retry.linkerd.io/timeout makes slow attempts retryable.

Enterprise Note

  • Retry rules belong to the team that runs the service – the route lives in the Service’s namespace and applies to all clients in the mesh.
  • Only one place with retries per chain, otherwise they multiply.
  • Remove ServiceProfiles from older installations before switching, otherwise the annotations do not take effect.
  • Alert on the ratio of outbound_http_route_retry_requests_total to all requests, not only on the success rate, which retries make look better than it is.
  • Let timeouts get shorter from the outside in: Ingress > web > backend.