Day 8: Circuit Breaking and Rate Limiting
Goal: A Pod that only returns errors hardly gets any traffic anymore, and a noisy client is throttled with 429 without disturbing the others. · Duration: approx. 60 min · Prerequisite: Day 7
Briefing
- In Linkerd, circuit breaking is called failure accrual: the client proxy counts errors per endpoint and takes a Pod out of the load balancer after N consecutive errors.
- It is controlled by annotations on the Service:
balancer.linkerd.io/failure-accrual: consecutiveplus...-max-failures,...-min-penalty,...-max-penalty. - After the penalty, a blocked Pod receives a real request as a probe. Green readiness probes do not bring it back.
- Rate limiting is done by the server proxy:
HTTPLocalRateLimitPolicyon aServer, withtotal,identityandoverrides. Too much yields 429. “Local” means: per Pod.
last ──> web ──(failure accrual)──> helper healthy
└──> helper healthy
└─X helper-kaputt (always 500, readiness green)
burst ──(too many requests)──> helper: 429 (HTTPLocalRateLimitPolicy)
Drill 1: Setup
Once again a copy in its own namespace. web only
forwards /get to helper, and the load
generator calls it three times per round.
mkdir -p ~/linkerd/tag08 && cd ~/linkerd/tag08
kubectl create namespace tag08
kubectl annotate namespace tag08 linkerd.io/inject=enabled
for a in web helper last; do kubectl -n tag08 create serviceaccount $a; done
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag08}
data:
default.conf: |
server {
listen 8080;
location /get {
proxy_pass http://helper.tag08.svc.cluster.local/get;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag08}
spec:
replicas: 1
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
serviceAccountName: web
containers:
- name: web
image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
ports: [{containerPort: 8080, name: http}]
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag08}
spec:
selector: {app: web}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: tag08}
spec:
replicas: 2
selector: {matchLabels: {app: helper, variante: gesund}}
template:
metadata: {labels: {app: helper, variante: gesund}}
spec:
serviceAccountName: helper
containers:
- name: helper
image: ghcr.io/mccutchen/go-httpbin:2.26.0
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: tag08}
spec:
selector: {app: helper}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag08}
spec:
replicas: 1
selector: {matchLabels: {app: last}}
template:
metadata: {labels: {app: last}}
spec:
serviceAccountName: last
containers:
- name: last
image: docker.io/curlimages/curl:8.23.0
command: [sh, -c]
args:
- |
while true; do
for p in /get /get /get; do
code=$(curl -s -o /dev/null -w "%{http_code}" -m 5 http://web$p)
echo "$(date +%T) $p $code"
done
sleep 1
done
EOF
kubectl apply -f app.yaml
kubectl -n tag08 rollout status deploy/helper --timeout=180s
kubectl -n tag08 rollout status deploy/last --timeout=180s
sleep 60
kubectl -n tag08 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -cExpected (numbers vary):
126 /get 200
The Service only selects app: helper – you will use the
label variante to tell the Pods apart in a moment.
Save your progress? Sign up for free
Drill 2: A Pod breaks
The worst case for a load balancer: a Pod is ready but only returns
errors. Here, an nginx that answers /status/200 (readiness)
and returns 500 for everything else:
cat > kaputt.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: kaputt-conf, namespace: tag08}
data:
default.conf: |
server {
listen 8080;
location = /status/200 { return 200 "ok\n"; }
location / { return 500 "kaputt\n"; }
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper-kaputt, namespace: tag08}
spec:
replicas: 1
selector: {matchLabels: {app: helper, variante: kaputt}}
template:
metadata: {labels: {app: helper, variante: kaputt}}
spec:
serviceAccountName: helper
containers:
- name: helper
image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /status/200, port: http}
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes: [{name: conf, configMap: {name: kaputt-conf}}]
EOF
kubectl apply -f kaputt.yaml
kubectl -n tag08 rollout status deploy/helper-kaputt --timeout=180s
sleep 65
kubectl -n tag08 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n tag08 po -l app=helper --from deploy/webExpected (abridged, numbers vary):
83 /get 200
43 /get 500
NAME STATUS MESHED SUCCESS RPS LATENCY_P50
helper-5897c7498b-skj58 Running 1/1 100.00% 0.8rps 0ms
helper-5897c7498b-xsxmm Running 1/1 100.00% 0.8rps 6ms
helper-kaputt-7c75dc7d79-pkcwx Running 1/1 0.00% 0.7rps 3ms
One third errors, evenly distributed: Kubernetes considers the Pod healthy, and Linkerd’s load balancer chooses by latency – the broken Pod even answers quickly.
Save your progress? Sign up for free
Drill 3: Turning on failure accrual
After three consecutive errors, the Pod should get a 10-second break, longer and longer with further errors, at most one minute:
kubectl -n tag08 annotate service helper \
balancer.linkerd.io/failure-accrual=consecutive \
balancer.linkerd.io/failure-accrual-consecutive-max-failures=3 \
balancer.linkerd.io/failure-accrual-consecutive-min-penalty=10s \
balancer.linkerd.io/failure-accrual-consecutive-max-penalty=1m
sleep 75
kubectl -n tag08 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n tag08 po -l app=helper --from deploy/webExpected (abridged, numbers vary):
service/helper annotated
124 /get 200
1 /get 500
NAME STATUS MESHED SUCCESS RPS LATENCY_P50
helper-5897c7498b-skj58 Running 1/1 100.00% 1.3rps 3ms
helper-5897c7498b-xsxmm Running 1/1 100.00% 0.9rps 8ms
helper-kaputt-7c75dc7d79-pkcwx Running 1/1 0.00% 0.0rps 0ms
From 34 to under 1 percent errors, without a restart and without a
change to web: every client proxy reads the annotation on
the Service. The broken Pod keeps running and stays in the Service –
only web no longer sends it anything.
Save your progress? Sign up for free
Drill 4: Probation
When does the Pod come back? The proxy of web counts
locked-out endpoints as pending:
linkerd diagnostics proxy-metrics -n tag08 deploy/web \
| grep '^outbound_http_balancer_endpoints.*pending' | grep -oE '[0-9]+$'
kubectl -n tag08 logs deploy/last -c last --since=120s | awk '$3==500 {print $1}' \
| tail -5Expected (times vary):
1
21:30:58
21:31:07
21:31:08
21:31:21
21:31:50
Until 21:31:08 the errors came thick and fast – that is how long the
annotation took to reach the proxy of web. Then one error
after 13 seconds, the next after 29: every probe request fails, and the
break grows (with some randomness) up to the limit of one minute. A few
minutes later:
kubectl -n tag08 logs deploy/last -c last --since=300s | awk '$3==500 {print $1}'Expected (times vary):
21:31:50
21:32:31
21:33:31
21:34:31
21:35:31
21:36:32
One probe request per minute, that is the price. It is a real user request – with a retry from Day 7 on the same route, the user does not notice that one either.
Warning: If all endpoints are blocked, the proxy answers with 503. Failure accrual protects against individual bad Pods, not against a broken release. And as with retries: a ServiceProfile for the Service turns it off.
Besides consecutive, there is unified:
additionally a minimum success rate within a time window
(balancer.alpha.linkerd.io/failure-accrual-success-rate-threshold).
That also catches Pods that lose only every second request and never
have three errors in a row.
Save your progress? Sign up for free
Drill 5: Rate limit for everyone
For rate limiting, you take out the broken Pod and bring in a noisy client:
kubectl -n tag08 delete deploy helper-kaputt
kubectl -n tag08 delete configmap kaputt-conf
kubectl -n tag08 create serviceaccount burst
kubectl -n tag08 create deployment burst --image=docker.io/curlimages/curl:8.23.0 \
-- sleep infinity
kubectl -n tag08 set serviceaccount deploy/burst burst
kubectl -n tag08 rollout status deploy/burst --timeout=120s
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 40); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -cExpected:
40 200
Around ten requests per second, all of them get through. A rate limit
is attached to a Server. No default-deny policy applies in
tag08, so you explicitly set accessPolicy to
all-unauthenticated – otherwise the new Server locks out
everything, as on Day 6.
cat > ratelimit.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: helper-http, namespace: tag08}
spec:
podSelector: {matchLabels: {app: helper}}
port: http
proxyProtocol: HTTP/1
accessPolicy: all-unauthenticated
---
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: helper-rl, namespace: tag08}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
total:
requestsPerSecond: 4
EOF
kubectl apply -f ratelimit.yaml
sleep 5
kubectl -n tag08 get httplocalratelimitpolicy helper-rl -o yaml | grep -A6 '^status'
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 40); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c
kubectl -n tag08 logs deploy/last -c last --since=10s | awk '{print $2, $3}' \
| sort | uniq -cExpected (abridged, numbers vary):
reason: Accepted
status: "True"
type: Accepted
18 200
22 429
15 /get 200
6 /get 429
burst is throttled – but so is last via
web. total is a shared pool: whoever comes
first uses it up. And it applies per Pod: two
helper Pods together let around 8 requests per second
through. The response comes from the proxy, without a body:
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 15); do
curl -si http://helper/get; done' | grep -m1 -A2 '^HTTP/1.1 429'
kubectl -n tag08 logs deploy/helper -c linkerd-proxy --since=1m \
| grep -o 'total rate limit exceeded: [0-9]*rps' | tail -1Expected:
HTTP/1.1 429 Too Many Requests
date: Thu, 19 Nov 2026 19:43:50 GMT
content-length: 0
total rate limit exceeded: 4rps
Save your progress? Sign up for free
Drill 6: Fair per identity
Each client should get at most 1 request per second and Pod,
web as an important customer 4 – via overrides
on its ServiceAccount. First attempt with total: 6:
sed -i 's/requestsPerSecond: 4$/requestsPerSecond: 6/' ratelimit.yaml
cat >> ratelimit.yaml <<'EOF'
identity:
requestsPerSecond: 1
overrides:
- requestsPerSecond: 4
clientRefs:
- {kind: ServiceAccount, name: web}
EOF
kubectl apply -f ratelimit.yaml
sleep 70
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 40); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c
kubectl -n tag08 logs deploy/last -c last --since=6s | awk '{print $2, $3}' \
| sort | uniq -c
for p in $(kubectl -n tag08 get pod -l app=helper -o name); do
linkerd diagnostics proxy-metrics -n tag08 $p | grep '^inbound_http_local_ratelimit' \
| grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z]+|tls="no_identity"|\} [0-9]+' \
| paste - - -
doneExpected (abridged, numbers vary):
6 200
34 429
10 /get 200
5 /get 429
ratelimit_scope="total" client_id="web } 11
ratelimit_scope="identity" client_id="burst } 18
ratelimit_scope="total" client_id="burst } 35
...
burst now hits the identity limit. web
still fails – in the total scope. Rejected requests from
burst also consume the shared pool. That is why
total is an emergency brake for the Pod, not a tool for
fairness: set it considerably higher.
cat > ratelimit-rl.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: helper-rl, namespace: tag08}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
total:
requestsPerSecond: 20
identity:
requestsPerSecond: 1
overrides:
- requestsPerSecond: 4
clientRefs:
- {kind: ServiceAccount, name: web}
EOF
kubectl apply -f ratelimit-rl.yaml
sleep 10
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 40); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c
kubectl -n tag08 logs deploy/last -c last --since=6s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag08 get httplocalratelimitpolicyExpected (numbers vary):
httplocalratelimitpolicy.policy.linkerd.io/helper-rl configured
4 200
36 429
15 /get 200
NAME TARGET_KIND TARGET_NAME TOTAL_RPS IDENTITY_RPS
helper-rl Server helper-http 20 1
burst stays throttled, last keeps running
undisturbed.
Save your progress? Sign up for free
Drill 7: Who gets how much?
Both clients at the same pace, around 5 requests per second:
kubectl -n tag08 exec deploy/web -c web -- sh -c 'for i in $(seq 30); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; sleep 0.2
done' | sort | uniq -c
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 30); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; sleep 0.2
done' | sort | uniq -cExpected (numbers vary):
27 200
3 429
13 200
17 429
web is allowed 4 per Pod; the few 429s occur because the
load generator is running through web at the same time.
burst gets only one request per second and Pod. The
assignment works via the mTLS identity. A client without a proxy has
none – it still falls under identity, shown in the metric
with tls="no_identity" instead of client_id.
In the lab, two such clients shared one limit.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the failure accrual annotations of the Service
helper. Target time: 20 s - Show how many endpoints the proxy of
webis currently locking out. Target time: 30 s - Show the rate limit policy with its target and limits. Target time: 15 s
- Show which clients failed at the limit in which scope. Target time: 45 s
Solution
kubectl -n tag08 describe service helper | grep -A4 Annotations
linkerd diagnostics proxy-metrics -n tag08 deploy/web \
| grep '^outbound_http_balancer_endpoints.*pending' | grep -oE '[0-9]+$'
kubectl -n tag08 get httplocalratelimitpolicy
linkerd diagnostics proxy-metrics -n tag08 deploy/helper \
| grep '^inbound_http_local_ratelimit' \
| grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z]+|tls="no_identity"|\} [0-9]+' \
| paste - - -Expected (abridged):
Annotations: balancer.linkerd.io/failure-accrual: consecutive
balancer.linkerd.io/failure-accrual-consecutive-max-failures: 3
balancer.linkerd.io/failure-accrual-consecutive-max-penalty: 1m
balancer.linkerd.io/failure-accrual-consecutive-min-penalty: 10s
0
NAME TARGET_KIND TARGET_NAME TOTAL_RPS IDENTITY_RPS
helper-rl Server helper-http 20 1
ratelimit_scope="total" client_id="burst } 37
ratelimit_scope="identity" client_id="burst } 73
ratelimit_scope="identity" client_id="web } 33
...
The broken Pod has been gone since Drill 5, hence 0
locked-out endpoints.
Save your progress? Sign up for free
Checkpoint
kubectl -n tag08 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -cExpected: only /get 200.
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in 1 2 3 4; do
curl -s -o /dev/null -w "%{http_code} " http://helper/get; done; echo'Expected: after one or two 200s,
429 too, for example 200 200 429 429.
Challenge
Bring the broken Pod back. It should receive a probe request at most every 10 seconds. Prove it in the load generator’s log.
Hint: max-penalty must be greater than
min-penalty.
Solution
kubectl apply -f kaputt.yaml
kubectl -n tag08 rollout status deploy/helper-kaputt --timeout=120s
kubectl -n tag08 annotate service helper --overwrite \
balancer.linkerd.io/failure-accrual-consecutive-min-penalty=5s \
balancer.linkerd.io/failure-accrual-consecutive-max-penalty=10s
sleep 75
kubectl -n tag08 logs deploy/last -c last --since=60s | awk '$3==500 {print $1}' \
| tr '\n' ' '; echoExpected (times vary):
19:50:24 19:50:34 19:50:45 19:50:56 19:51:06 19:51:19
One error roughly every 10 seconds – the break no longer grows beyond the limit. Shorter penalties bring a repaired Pod back faster and cost more probe errors.
Save your progress? Sign up for free
Cleanup
kubectl delete namespace tag08Interview Note
- How does Linkerd do circuit breaking? Failure accrual in the client proxy, per endpoint; after N consecutive errors, a lockout with exponential backoff, return via real probe requests. Configured by annotation on the Service.
- Why is the readiness probe not enough? It checks one path, not the behavior with real requests; a Pod can be ready and still be broken.
- Where does the rate limit take effect? In the server proxy
of each Pod – local, not cluster-wide.
identityensures fairness,totalis the emergency brake.
Enterprise Note
- Plan failure accrual and retries together: retries bridge the probe errors, failure accrual prevents retries from continuing to feed a broken Pod.
- Rate limits are per Pod: when you scale, the overall limit rises with it. For hard global limits (API quotas), you need a central rate limit at the gateway.
- Clients without a proxy have no identity and share one pool in the
identityscope – one of them can starve the others. - Alert on
inbound_http_local_ratelimit_totalperclient_id: it shows who is being throttled before a team complains about 429s. - With
unified, a 429 counts as an error: a rate limit can then throw healthy Pods out of the load balancer. Test both together.