Day 7: Retries and Timeouts
Goal:
/flakysucceeds almost 100 percent of the time with retries,/slowaborts after one second – and you know what retries cost and when they are dangerous. · Duration: approx. 60 min · Prerequisite: Day 6
Briefing
- Retries and timeouts are handled by the client proxy (outbound). The app does not notice anything.
- You configure them through a Gateway API HTTPRoute
with the Service as its
parentRefand annotations:retry.linkerd.io/http,retry.linkerd.io/limit,retry.linkerd.io/timeout,timeout.linkerd.io/request(orresponse,idle). - Once a route is attached to a Service, only what a route matches still applies to it.
- A retry is an additional request: more load, and with
POSTpossibly a duplicate order.
last ──> web ──(outbound proxy: route, retry, timeout)──> helper (2 Pods)
/unstable retry on 5xx, max. 2 retries
/delay timeout 1 s
Drill 1: A copy for experiments
notes stays untouched. In tag07, a small
copy of web, helper and last runs
with its own ServiceAccounts. Here the load generator calls
/flaky twice and /slow once.
mkdir -p ~/linkerd/tag07 && cd ~/linkerd/tag07
kubectl create namespace tag07
kubectl annotate namespace tag07 linkerd.io/inject=enabled
for a in web helper last; do kubectl -n tag07 create serviceaccount $a; done
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag07}
data:
default.conf: |
server {
listen 8080;
location /flaky {
proxy_pass http://helper.tag07.svc.cluster.local/unstable?failure_rate=0.3;
}
location /slow {
proxy_pass http://helper.tag07.svc.cluster.local/delay/2;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag07}
spec:
replicas: 1
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
serviceAccountName: web
containers:
- name: web
image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
ports: [{containerPort: 8080, name: http}]
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag07}
spec:
selector: {app: web}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: tag07}
spec:
replicas: 2
selector: {matchLabels: {app: helper}}
template:
metadata: {labels: {app: helper}}
spec:
serviceAccountName: helper
containers:
- name: helper
image: ghcr.io/mccutchen/go-httpbin:2.26.0
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: tag07}
spec:
selector: {app: helper}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag07}
spec:
replicas: 1
selector: {matchLabels: {app: last}}
template:
metadata: {labels: {app: last}}
spec:
serviceAccountName: last
containers:
- name: last
image: docker.io/curlimages/curl:8.23.0
command: [sh, -c]
args:
- |
while true; do
for p in /flaky /flaky /slow; do
code=$(curl -s -o /dev/null -w "%{http_code}" -m 5 http://web$p)
echo "$(date +%T) $p $code"
done
sleep 1
done
EOF
kubectl apply -f app.yaml
kubectl -n tag07 rollout status deploy/helper --timeout=180s
kubectl -n tag07 get podsExpected (abridged):
NAME READY STATUS RESTARTS AGE
helper-79d57fbbbb-cxfnr 2/2 Running 0 13s
helper-79d57fbbbb-wknd2 2/2 Running 0 14s
last-84d85f57b9-rbfjf 2/2 Running 0 14s
web-7d94ccc84d-bgq62 2/2 Running 0 14s
For measuring per route there is
linkerd viz stat-outbound. It queries Prometheus
from your machine – with the Prometheus from
monitoring, that means through a port-forward that you
leave open for today:
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
PROM=--prometheusURL=http://localhost:9090
sleep 60
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat-outbound -n tag07 deploy/web $PROMExpected (abridged, numbers vary):
20 /flaky 200
14 /flaky 500
17 /slow 200
NAME SERVICE ROUTE TYPE BACKEND SUCCESS RPS ... TIMEOUTS RETRIES
web helper:80 [default] 76.00% 0.83 ... 0.00% 0.00%
└─────────────► helper:80 76.00% 0.83 ... 0.00%
Without $PROM, the CLI tries to resolve the Service name
from the viz configuration directly and fails with
lookup kps-kube-prometheus-stack-prometheus... server misbehaving.
[default] is the route Linkerd assumes when there is no
HTTPRoute of your own.
Save your progress? Sign up for free
Drill 2: Retries for /unstable
cat > route-retry.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: helper-unstable
namespace: tag07
annotations:
retry.linkerd.io/http: 5xx
retry.linkerd.io/limit: "2"
spec:
parentRefs:
- {group: core, kind: Service, name: helper, port: 80}
rules:
- matches:
- path: {type: PathPrefix, value: /unstable}
EOF
kubectl apply -f route-retry.yaml
kubectl -n tag07 get httproute helper-unstable -o yaml | grep -E 'controllerName|reason'
sleep 70
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat-outbound -n tag07 deploy/web $PROMExpected (abridged, numbers vary):
reason: Accepted
reason: ResolvedRefs
controllerName: linkerd.io/policy-controller
83 /flaky 200
1 /flaky 500
42 /slow 404
ROUTE TYPE BACKEND SUCCESS RPS ... TIMEOUTS RETRIES
helper-unstable HTTPRoute 100.00% 1.35 ... 0.00% 27.03%
└────────────────────────► helper:80 72.32% 1.87 ... 0.00%
Warning: Cilium reads HTTPRoutes too, and for a route with a Service as its
parentRefit creates an EndpointSlicecilium-gateway-<service>with the placeholder address192.192.192.192:9999. It belongs to no real Service and does not interfere with Linkerd, but it stays behind after you delete the route. Only Linkerd sets the route’s status. To show it:kubectl get endpointslices -A | grep cilium-gateway.
/flaky jumps from around 70 to almost 100 percent: three
attempts fail only with a probability of 0.3 × 0.3 × 0.3 ≈ 3 percent.
The route’s line shows what web experiences, the line below
it what helper really delivers – still 72 percent. And
/slow?
Save your progress? Sign up for free
Drill 3: The trap with the remaining paths
/slow suddenly returns 404, faster than
before. As soon as an HTTPRoute is attached to the Service, it replaces
the default route. The client proxy itself answers paths that no rule
matches with 404. So you need a route for the rest:
cat > route-rest.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: helper-rest, namespace: tag07}
spec:
parentRefs:
- {group: core, kind: Service, name: helper, port: 80}
rules:
- matches:
- path: {type: PathPrefix, value: /}
EOF
kubectl apply -f route-rest.yaml
sleep 70
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat-outbound -n tag07 deploy/web $PROM | grep -v '└'Expected (abridged):
36 /flaky 200
17 /slow 200
ROUTE TYPE BACKEND SUCCESS RPS ... TIMEOUTS RETRIES
helper-unstable HTTPRoute 100.00% 0.55 ... 0.00% 26.67%
helper-rest HTTPRoute 100.00% 0.30 ... 0.00% 0.00%
The Gateway API picks the most specific rule: /unstable
wins over /.
Warning: In
notes, the same mistake would have taken down every path except that one. Whoever attaches the first route to a Service always adds the one for/right along with it.
Save your progress? Sign up for free
Drill 4: The price
You already saw it in Drill 2: web sends 1.35 requests
per second, helper receives 1.87. Retries are load. How bad
it gets when helper fails completely is shown by a call
directly from last – its proxy knows the same route:
kubectl -n tag07 exec deploy/last -c last -- sh -c 'for i in $(seq 20); do
curl -s -o /dev/null -w "%{http_code}\n" "http://helper/unstable?failure_rate=1"
done' | sort | uniq -c
linkerd diagnostics proxy-metrics -n tag07 deploy/last \
| grep -E '^outbound_http_route_retry_(requests|limit_exceeded)_total' \
| sed -E 's/\{.*\}//'Expected:
20 500
outbound_http_route_retry_limit_exceeded_total 20
outbound_http_route_retry_requests_total 40
last never called helper directly before,
so the counters belong to this test only. 20 requests, 40 retries:
helper sees three times the load – exactly when it is
already down. In chains, this multiplies: three services with two
retries each turn one request into up to 27.
In the past, a retry budget in the ServiceProfile
limited this (retryRatio: 0.2 meant: at most 20 percent
additional load). The annotations on HTTPRoutes know no budget, only
limit. Keep it small and rely on circuit breaking (Day 8)
against persistent failures. You cannot mix the two: if a ServiceProfile
exists for a Service, the proxy ignores the annotations.
Save your progress? Sign up for free
Drill 5: Timeouts
/slow hangs for two seconds. Your goal: give up after
one second.
cat > route-delay.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: helper-delay
namespace: tag07
annotations:
timeout.linkerd.io/request: 1s
spec:
parentRefs:
- {group: core, kind: Service, name: helper, port: 80}
rules:
- matches:
- path: {type: PathPrefix, value: /delay}
EOF
kubectl apply -f route-delay.yaml
sleep 5
kubectl -n tag07 exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://web/slow
kubectl -n tag07 logs deploy/web -c linkerd-proxy --since=30s \
| grep -o 'route HTTPRoute[^:]*: [^,]*request timeout: 1s' | tail -1Expected (abridged):
504 1.019408s
route HTTPRoute.tag07.helper-delay: backend Service.tag07.helper:80: ...
The proxy of web aborts after one second and answers
504 Gateway Timeout, which nginx passes through. After one
minute:
sleep 60
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat-outbound -n tag07 deploy/web $PROM | grep helper-delayExpected (abridged):
49 /flaky 200
1 /flaky 500
24 /slow 504
helper-delay HTTPRoute 0.00% 0.42 ... 100.00% 0.00%
The TIMEOUTS column shows 100 percent. The latency
columns in stat-outbound say little for timeouts – they are
estimated from coarse histogram buckets. A timeout is
not retried. If you want that, set
retry.linkerd.io/timeout (time per attempt) together with
retry.linkerd.io/http.
Save your progress? Sign up for free
Drill 6: POST is not idempotent
The route helper-unstable matches every method. What
happens to an order?
timeout 15 linkerd viz tap -n tag07 deploy/helper --method POST > tap-post.txt &
sleep 5
kubectl -n tag07 exec deploy/last -c last -- curl -s -o /dev/null -w '%{http_code}\n' \
-X POST -d 'bestellung=1' 'http://helper/unstable?failure_rate=1'
sleep 12
grep -c '^req' tap-post.txtExpected:
500
3
One POST from the client, three at helper. Had the first
attempt stored the order and only then answered with 500, it would now
exist three times. Linkerd does not check the method – it only never
retries bodies over 64 KiB. So only GET:
sed -i 's| - path: {type: PathPrefix, value: /unstable}|&\n method: GET|' \
route-retry.yaml
tail -3 route-retry.yaml
kubectl apply -f route-retry.yaml
sleep 5
timeout 15 linkerd viz tap -n tag07 deploy/helper --method POST > tap-post.txt &
sleep 5
kubectl -n tag07 exec deploy/last -c last -- curl -s -o /dev/null -w '%{http_code}\n' \
-X POST -d 'bestellung=1' 'http://helper/unstable?failure_rate=1'
sleep 12
grep -c '^req' tap-post.txtExpected:
- matches:
- path: {type: PathPrefix, value: /unstable}
method: GET
httproute.gateway.networking.k8s.io/helper-unstable configured
500
1
The POST now falls through to helper-rest without
retries. GET, HEAD, PUT and
DELETE are idempotent according to HTTP – whether your app
implements them that way too is something you check before you turn on
retries.
Save your progress? Sign up for free
Drill 7: Retries in Prometheus
For dashboards and alerts, you count directly:
Q='sum by (route_name) (rate(outbound_http_route_retry_requests_total{
namespace="tag07", deployment="web"}[2m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
| jq -r '.data.result[] | .metric.route_name + " " + .value[1]'
Q='sum by (route_name, http_status) (rate(outbound_http_route_request_statuses_total{
namespace="tag07", deployment="web"}[2m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
| jq -r '.data.result[] | .metric.route_name + " " + .metric.http_status
+ " " + .value[1]' | grep -v ' 0$'Expected (abridged, numbers vary):
helper-unstable 0.23636363636363636
helper-unstable 200 0.7999999999999999
helper-delay 0.3818181818181818
helper-delay 200 0.027166000000000003
Aborted requests have no http_status – the gap at
helper-delay is the timeouts. A rising ratio of retries to
requests is an early alarm: the service behind it is getting worse, even
if users do not notice anything yet.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all HTTPRoutes in
tag07with their Linkerd annotations. Target time: 30 s - Raise the retry limit of
helper-unstableto 3 without changing the file. Target time: 30 s - Show how many retries the proxy of
webhas sent so far. Target time: 30 s - Show whether the policy controller has accepted all routes. Target time: 45 s
Solution
kubectl -n tag07 describe httproute | grep -E '^Name:|linkerd.io/'
kubectl -n tag07 annotate httproute helper-unstable retry.linkerd.io/limit=3 --overwrite
linkerd diagnostics proxy-metrics -n tag07 deploy/web \
| grep '^outbound_http_route_retry_requests_total' | grep -oE '[0-9]+$'
kubectl -n tag07 get httproute \
-o custom-columns=NAME:.metadata.name,STATUS:.status.parents[0].conditions[0].reasonExpected (abridged):
Name: helper-delay
Annotations: timeout.linkerd.io/request: 1s
...
Name: helper-unstable
Annotations: retry.linkerd.io/http: 5xx
retry.linkerd.io/limit: 2
httproute.gateway.networking.k8s.io/helper-unstable annotated
145
NAME STATUS
helper-delay Accepted
helper-rest Accepted
helper-unstable Accepted
Save your progress? Sign up for free
Checkpoint
kubectl -n tag07 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
| sort | uniq -cExpected: /flaky almost only
200, /slow only 504.
48 /flaky 200
25 /slow 504
linkerd viz stat-outbound -n tag07 deploy/web $PROM | grep -c HTTPRouteExpected: 2 –
helper-unstable and helper-delay. Nothing from
the load generator goes through helper-rest anymore, since
/delay has its own route.
Challenge
Change the timeout for /delay to three seconds – this
time using the Gateway API field instead of the annotation – and find
out what applies when both are set.
Hint: The field is called
timeouts.request and sits in the rule
(rules[]).
Solution
kubectl -n tag07 annotate httproute helper-delay timeout.linkerd.io/request-
kubectl -n tag07 patch httproute helper-delay --type=json \
-p '[{"op":"add","path":"/spec/rules/0/timeouts","value":{"request":"3s"}}]'
sleep 5
kubectl -n tag07 exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://web/slow
kubectl -n tag07 annotate httproute helper-delay timeout.linkerd.io/request=1s
sleep 5
kubectl -n tag07 exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://web/slowExpected:
httproute.gateway.networking.k8s.io/helper-delay annotated
httproute.gateway.networking.k8s.io/helper-delay patched
200 2.026916s
httproute.gateway.networking.k8s.io/helper-delay annotated
504 1.039460s
With 3 s, /slow gets through again. If both the field
and the annotation are set, the annotation wins. For
response and idle timeouts, only the
annotations exist.
Save your progress? Sign up for free
Cleanup
kill %1
kubectl delete namespace tag07
kubectl get endpointslices -A | grep cilium-gatewayExpected: no line from tag07 anymore –
Cilium’s EndpointSlice disappears along with the namespace. If you only
delete a route, you remove it yourself:
kubectl -n <ns> delete endpointslice cilium-gateway-<service>.
Interview Note
- Where do retries run in Linkerd? In the client proxy,
configured via HTTPRoute or Service annotation
(
retry.linkerd.io/*). The app does not notice anything. - Why not retries everywhere? They multiply the load exactly when things fail and are dangerous with non-idempotent methods such as POST.
- Timeout or retry first? Both: the timeout limits the user’s
waiting time,
retry.linkerd.io/timeoutmakes slow attempts retryable.
Enterprise Note
- Retry rules belong to the team that runs the service – the route lives in the Service’s namespace and applies to all clients in the mesh.
- Only one place with retries per chain, otherwise they multiply.
- Remove ServiceProfiles from older installations before switching, otherwise the annotations do not take effect.
- Alert on the ratio of
outbound_http_route_retry_requests_totalto all requests, not only on the success rate, which retries make look better than it is. - Let timeouts get shorter from the outside in: Ingress > web > backend.