Skip to content
Hands-on IT IT Drills

Day 8: Circuit Breaking and Rate Limiting

Goal: A Pod that only returns errors hardly gets any traffic anymore, and a noisy client is throttled with 429 without disturbing the others. · Duration: approx. 60 min · Prerequisite: Day 7

Briefing

  • In Linkerd, circuit breaking is called failure accrual: the client proxy counts errors per endpoint and takes a Pod out of the load balancer after N consecutive errors.
  • It is controlled by annotations on the Service: balancer.linkerd.io/failure-accrual: consecutive plus ...-max-failures, ...-min-penalty, ...-max-penalty.
  • After the penalty, a blocked Pod receives a real request as a probe. Green readiness probes do not bring it back.
  • Rate limiting is done by the server proxy: HTTPLocalRateLimitPolicy on a Server, with total, identity and overrides. Too much yields 429. “Local” means: per Pod.
 last ──> web ──(failure accrual)──> helper  healthy
                                └──> helper  healthy
                                └─X  helper-kaputt (always 500, readiness green)
 burst ──(too many requests)──> helper: 429 (HTTPLocalRateLimitPolicy)

Drill 1: Setup

Once again a copy in its own namespace. web only forwards /get to helper, and the load generator calls it three times per round.

mkdir -p ~/linkerd/tag08 && cd ~/linkerd/tag08
kubectl create namespace tag08
kubectl annotate namespace tag08 linkerd.io/inject=enabled
for a in web helper last; do kubectl -n tag08 create serviceaccount $a; done
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag08}
data:
  default.conf: |
    server {
      listen 8080;
      location /get {
        proxy_pass http://helper.tag08.svc.cluster.local/get;
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag08}
spec:
  replicas: 1
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      serviceAccountName: web
      containers:
      - name: web
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag08}
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: tag08}
spec:
  replicas: 2
  selector: {matchLabels: {app: helper, variante: gesund}}
  template:
    metadata: {labels: {app: helper, variante: gesund}}
    spec:
      serviceAccountName: helper
      containers:
      - name: helper
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: tag08}
spec:
  selector: {app: helper}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag08}
spec:
  replicas: 1
  selector: {matchLabels: {app: last}}
  template:
    metadata: {labels: {app: last}}
    spec:
      serviceAccountName: last
      containers:
      - name: last
        image: docker.io/curlimages/curl:8.23.0
        command: [sh, -c]
        args:
        - |
          while true; do
            for p in /get /get /get; do
              code=$(curl -s -o /dev/null -w "%{http_code}" -m 5 http://web$p)
              echo "$(date +%T) $p $code"
            done
            sleep 1
          done
EOF
kubectl apply -f app.yaml
kubectl -n tag08 rollout status deploy/helper --timeout=180s
kubectl -n tag08 rollout status deploy/last --timeout=180s
sleep 60
kubectl -n tag08 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (numbers vary):

    126 /get 200

The Service only selects app: helper – you will use the label variante to tell the Pods apart in a moment.

Save your progress? Sign up for free

Drill 2: A Pod breaks

The worst case for a load balancer: a Pod is ready but only returns errors. Here, an nginx that answers /status/200 (readiness) and returns 500 for everything else:

cat > kaputt.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: kaputt-conf, namespace: tag08}
data:
  default.conf: |
    server {
      listen 8080;
      location = /status/200 { return 200 "ok\n"; }
      location / { return 500 "kaputt\n"; }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper-kaputt, namespace: tag08}
spec:
  replicas: 1
  selector: {matchLabels: {app: helper, variante: kaputt}}
  template:
    metadata: {labels: {app: helper, variante: kaputt}}
    spec:
      serviceAccountName: helper
      containers:
      - name: helper
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /status/200, port: http}
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: kaputt-conf}}]
EOF
kubectl apply -f kaputt.yaml
kubectl -n tag08 rollout status deploy/helper-kaputt --timeout=180s
sleep 65
kubectl -n tag08 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n tag08 po -l app=helper --from deploy/web

Expected (abridged, numbers vary):

     83 /get 200
     43 /get 500
NAME                              STATUS   MESHED   SUCCESS      RPS   LATENCY_P50
helper-5897c7498b-skj58          Running      1/1   100.00%   0.8rps           0ms
helper-5897c7498b-xsxmm          Running      1/1   100.00%   0.8rps           6ms
helper-kaputt-7c75dc7d79-pkcwx   Running      1/1     0.00%   0.7rps           3ms

One third errors, evenly distributed: Kubernetes considers the Pod healthy, and Linkerd’s load balancer chooses by latency – the broken Pod even answers quickly.

Save your progress? Sign up for free

Drill 3: Turning on failure accrual

After three consecutive errors, the Pod should get a 10-second break, longer and longer with further errors, at most one minute:

kubectl -n tag08 annotate service helper \
  balancer.linkerd.io/failure-accrual=consecutive \
  balancer.linkerd.io/failure-accrual-consecutive-max-failures=3 \
  balancer.linkerd.io/failure-accrual-consecutive-min-penalty=10s \
  balancer.linkerd.io/failure-accrual-consecutive-max-penalty=1m
sleep 75
kubectl -n tag08 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n tag08 po -l app=helper --from deploy/web

Expected (abridged, numbers vary):

service/helper annotated
    124 /get 200
      1 /get 500
NAME                              STATUS   MESHED   SUCCESS      RPS   LATENCY_P50
helper-5897c7498b-skj58          Running      1/1   100.00%   1.3rps           3ms
helper-5897c7498b-xsxmm          Running      1/1   100.00%   0.9rps           8ms
helper-kaputt-7c75dc7d79-pkcwx   Running      1/1     0.00%   0.0rps           0ms

From 34 to under 1 percent errors, without a restart and without a change to web: every client proxy reads the annotation on the Service. The broken Pod keeps running and stays in the Service – only web no longer sends it anything.

Save your progress? Sign up for free

Drill 4: Probation

When does the Pod come back? The proxy of web counts locked-out endpoints as pending:

linkerd diagnostics proxy-metrics -n tag08 deploy/web \
  | grep '^outbound_http_balancer_endpoints.*pending' | grep -oE '[0-9]+$'
kubectl -n tag08 logs deploy/last -c last --since=120s | awk '$3==500 {print $1}' \
  | tail -5

Expected (times vary):

1
21:30:58
21:31:07
21:31:08
21:31:21
21:31:50

Until 21:31:08 the errors came thick and fast – that is how long the annotation took to reach the proxy of web. Then one error after 13 seconds, the next after 29: every probe request fails, and the break grows (with some randomness) up to the limit of one minute. A few minutes later:

kubectl -n tag08 logs deploy/last -c last --since=300s | awk '$3==500 {print $1}'

Expected (times vary):

21:31:50
21:32:31
21:33:31
21:34:31
21:35:31
21:36:32

One probe request per minute, that is the price. It is a real user request – with a retry from Day 7 on the same route, the user does not notice that one either.

Warning: If all endpoints are blocked, the proxy answers with 503. Failure accrual protects against individual bad Pods, not against a broken release. And as with retries: a ServiceProfile for the Service turns it off.

Besides consecutive, there is unified: additionally a minimum success rate within a time window (balancer.alpha.linkerd.io/failure-accrual-success-rate-threshold). That also catches Pods that lose only every second request and never have three errors in a row.

Save your progress? Sign up for free

Drill 5: Rate limit for everyone

For rate limiting, you take out the broken Pod and bring in a noisy client:

kubectl -n tag08 delete deploy helper-kaputt
kubectl -n tag08 delete configmap kaputt-conf
kubectl -n tag08 create serviceaccount burst
kubectl -n tag08 create deployment burst --image=docker.io/curlimages/curl:8.23.0 \
  -- sleep infinity
kubectl -n tag08 set serviceaccount deploy/burst burst
kubectl -n tag08 rollout status deploy/burst --timeout=120s
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 40); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c

Expected:

     40 200

Around ten requests per second, all of them get through. A rate limit is attached to a Server. No default-deny policy applies in tag08, so you explicitly set accessPolicy to all-unauthenticated – otherwise the new Server locks out everything, as on Day 6.

cat > ratelimit.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: helper-http, namespace: tag08}
spec:
  podSelector: {matchLabels: {app: helper}}
  port: http
  proxyProtocol: HTTP/1
  accessPolicy: all-unauthenticated
---
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: helper-rl, namespace: tag08}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
  total:
    requestsPerSecond: 4
EOF
kubectl apply -f ratelimit.yaml
sleep 5
kubectl -n tag08 get httplocalratelimitpolicy helper-rl -o yaml | grep -A6 '^status'
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 40); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c
kubectl -n tag08 logs deploy/last -c last --since=10s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (abridged, numbers vary):

    reason: Accepted
    status: "True"
    type: Accepted
     18 200
     22 429
     15 /get 200
      6 /get 429

burst is throttled – but so is last via web. total is a shared pool: whoever comes first uses it up. And it applies per Pod: two helper Pods together let around 8 requests per second through. The response comes from the proxy, without a body:

kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 15); do
  curl -si http://helper/get; done' | grep -m1 -A2 '^HTTP/1.1 429'
kubectl -n tag08 logs deploy/helper -c linkerd-proxy --since=1m \
  | grep -o 'total rate limit exceeded: [0-9]*rps' | tail -1

Expected:

HTTP/1.1 429 Too Many Requests
date: Thu, 19 Nov 2026 19:43:50 GMT
content-length: 0
total rate limit exceeded: 4rps

Save your progress? Sign up for free

Drill 6: Fair per identity

Each client should get at most 1 request per second and Pod, web as an important customer 4 – via overrides on its ServiceAccount. First attempt with total: 6:

sed -i 's/requestsPerSecond: 4$/requestsPerSecond: 6/' ratelimit.yaml
cat >> ratelimit.yaml <<'EOF'
  identity:
    requestsPerSecond: 1
  overrides:
  - requestsPerSecond: 4
    clientRefs:
    - {kind: ServiceAccount, name: web}
EOF
kubectl apply -f ratelimit.yaml
sleep 70
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 40); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c
kubectl -n tag08 logs deploy/last -c last --since=6s | awk '{print $2, $3}' \
  | sort | uniq -c
for p in $(kubectl -n tag08 get pod -l app=helper -o name); do
  linkerd diagnostics proxy-metrics -n tag08 $p | grep '^inbound_http_local_ratelimit' \
    | grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z]+|tls="no_identity"|\} [0-9]+' \
    | paste - - -
done

Expected (abridged, numbers vary):

      6 200
     34 429
     10 /get 200
      5 /get 429
ratelimit_scope="total"     client_id="web    } 11
ratelimit_scope="identity"  client_id="burst  } 18
ratelimit_scope="total"     client_id="burst  } 35
...

burst now hits the identity limit. web still fails – in the total scope. Rejected requests from burst also consume the shared pool. That is why total is an emergency brake for the Pod, not a tool for fairness: set it considerably higher.

cat > ratelimit-rl.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: helper-rl, namespace: tag08}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
  total:
    requestsPerSecond: 20
  identity:
    requestsPerSecond: 1
  overrides:
  - requestsPerSecond: 4
    clientRefs:
    - {kind: ServiceAccount, name: web}
EOF
kubectl apply -f ratelimit-rl.yaml
sleep 10
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 40); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c
kubectl -n tag08 logs deploy/last -c last --since=6s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag08 get httplocalratelimitpolicy

Expected (numbers vary):

httplocalratelimitpolicy.policy.linkerd.io/helper-rl configured
      4 200
     36 429
     15 /get 200
NAME        TARGET_KIND   TARGET_NAME   TOTAL_RPS   IDENTITY_RPS
helper-rl   Server        helper-http   20          1

burst stays throttled, last keeps running undisturbed.

Save your progress? Sign up for free

Drill 7: Who gets how much?

Both clients at the same pace, around 5 requests per second:

kubectl -n tag08 exec deploy/web -c web -- sh -c 'for i in $(seq 30); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; sleep 0.2
  done' | sort | uniq -c
kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in $(seq 30); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; sleep 0.2
  done' | sort | uniq -c

Expected (numbers vary):

     27 200
      3 429
     13 200
     17 429

web is allowed 4 per Pod; the few 429s occur because the load generator is running through web at the same time. burst gets only one request per second and Pod. The assignment works via the mTLS identity. A client without a proxy has none – it still falls under identity, shown in the metric with tls="no_identity" instead of client_id. In the lab, two such clients shared one limit.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the failure accrual annotations of the Service helper. Target time: 20 s
  2. Show how many endpoints the proxy of web is currently locking out. Target time: 30 s
  3. Show the rate limit policy with its target and limits. Target time: 15 s
  4. Show which clients failed at the limit in which scope. Target time: 45 s

Solution

kubectl -n tag08 describe service helper | grep -A4 Annotations
linkerd diagnostics proxy-metrics -n tag08 deploy/web \
  | grep '^outbound_http_balancer_endpoints.*pending' | grep -oE '[0-9]+$'
kubectl -n tag08 get httplocalratelimitpolicy
linkerd diagnostics proxy-metrics -n tag08 deploy/helper \
  | grep '^inbound_http_local_ratelimit' \
  | grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z]+|tls="no_identity"|\} [0-9]+' \
  | paste - - -

Expected (abridged):

Annotations:   balancer.linkerd.io/failure-accrual: consecutive
               balancer.linkerd.io/failure-accrual-consecutive-max-failures: 3
               balancer.linkerd.io/failure-accrual-consecutive-max-penalty: 1m
               balancer.linkerd.io/failure-accrual-consecutive-min-penalty: 10s
0
NAME        TARGET_KIND   TARGET_NAME   TOTAL_RPS   IDENTITY_RPS
helper-rl   Server        helper-http   20          1
ratelimit_scope="total"     client_id="burst  } 37
ratelimit_scope="identity"  client_id="burst  } 73
ratelimit_scope="identity"  client_id="web    } 33
...

The broken Pod has been gone since Drill 5, hence 0 locked-out endpoints.

Save your progress? Sign up for free

Checkpoint

kubectl -n tag08 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected: only /get 200.

kubectl -n tag08 exec deploy/burst -c curl -- sh -c 'for i in 1 2 3 4; do
  curl -s -o /dev/null -w "%{http_code} " http://helper/get; done; echo'

Expected: after one or two 200s, 429 too, for example 200 200 429 429.

Challenge

Bring the broken Pod back. It should receive a probe request at most every 10 seconds. Prove it in the load generator’s log.

Hint: max-penalty must be greater than min-penalty.

Solution

kubectl apply -f kaputt.yaml
kubectl -n tag08 rollout status deploy/helper-kaputt --timeout=120s
kubectl -n tag08 annotate service helper --overwrite \
  balancer.linkerd.io/failure-accrual-consecutive-min-penalty=5s \
  balancer.linkerd.io/failure-accrual-consecutive-max-penalty=10s
sleep 75
kubectl -n tag08 logs deploy/last -c last --since=60s | awk '$3==500 {print $1}' \
  | tr '\n' ' '; echo

Expected (times vary):

19:50:24 19:50:34 19:50:45 19:50:56 19:51:06 19:51:19

One error roughly every 10 seconds – the break no longer grows beyond the limit. Shorter penalties bring a repaired Pod back faster and cost more probe errors.

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag08

Interview Note

  • How does Linkerd do circuit breaking? Failure accrual in the client proxy, per endpoint; after N consecutive errors, a lockout with exponential backoff, return via real probe requests. Configured by annotation on the Service.
  • Why is the readiness probe not enough? It checks one path, not the behavior with real requests; a Pod can be ready and still be broken.
  • Where does the rate limit take effect? In the server proxy of each Pod – local, not cluster-wide. identity ensures fairness, total is the emergency brake.

Enterprise Note

  • Plan failure accrual and retries together: retries bridge the probe errors, failure accrual prevents retries from continuing to feed a broken Pod.
  • Rate limits are per Pod: when you scale, the overall limit rises with it. For hard global limits (API quotas), you need a central rate limit at the gateway.
  • Clients without a proxy have no identity and share one pool in the identity scope – one of them can starve the others.
  • Alert on inbound_http_local_ratelimit_total per client_id: it shows who is being throttled before a team complains about 429s.
  • With unified, a 429 counts as an error: a rate limit can then throw healthy Pods out of the load balancer. Test both together.