Skip to content
Hands-on IT IT Drills

Day 17: What the Mesh Costs

Goal: You measure for yourself how much latency, CPU and memory the proxy costs, and control its resources via annotations. · Duration: approx. 90 min · Prerequisite: Day 4

Briefing

  • Every request in the mesh passes through two proxies: the client’s and the server’s. That costs latency and CPU on both sides.
  • Do not trust any benchmark, not even one from this book: build the same setup twice, once without and once with a proxy, and measure at a fixed rate.
  • Lab measurements fluctuate. Run several times, take the median, state the range.
  • You set the proxy’s resources with config.linkerd.io/proxy-cpu-request and its relatives. Without them, the proxy is BestEffort.
 worker1                              worker2
 last-ohne (fortio) ───────────────> server-ohne (go-httpbin)
 last-mit  (fortio) ─> proxy ═mTLS═> proxy ─> server-mit (go-httpbin)

Drill 1: Two identical setups

The namespace stays without injection. Only the two -mit Deployments get the proxy via a Pod annotation. nodeSelector keeps clients and servers on fixed nodes; otherwise you end up comparing nodes instead of proxies.

mkdir -p ~/linkerd/tag17 && cd ~/linkerd/tag17
kubectl create namespace tag17
cat > app.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: server-ohne, namespace: tag17}
spec:
  replicas: 1
  selector: {matchLabels: {app: server-ohne}}
  template:
    metadata: {labels: {app: server-ohne}}
    spec:
      nodeSelector: {kubernetes.io/hostname: worker2}
      containers:
      - name: httpbin
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        ports: [{containerPort: 8080, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: server-mit, namespace: tag17}
spec:
  replicas: 1
  selector: {matchLabels: {app: server-mit}}
  template:
    metadata:
      labels: {app: server-mit}
      annotations: {linkerd.io/inject: enabled}
    spec:
      nodeSelector: {kubernetes.io/hostname: worker2}
      containers:
      - name: httpbin
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        ports: [{containerPort: 8080, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last-ohne, namespace: tag17}
spec:
  replicas: 1
  selector: {matchLabels: {app: last-ohne}}
  template:
    metadata: {labels: {app: last-ohne}}
    spec:
      nodeSelector: {kubernetes.io/hostname: worker1}
      containers:
      - name: fortio
        image: docker.io/fortio/fortio:1.69.5
        args: [server]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last-mit, namespace: tag17}
spec:
  replicas: 1
  selector: {matchLabels: {app: last-mit}}
  template:
    metadata:
      labels: {app: last-mit}
      annotations: {linkerd.io/inject: enabled}
    spec:
      nodeSelector: {kubernetes.io/hostname: worker1}
      containers:
      - name: fortio
        image: docker.io/fortio/fortio:1.69.5
        args: [server]
EOF
kubectl apply -f app.yaml
kubectl -n tag17 create service clusterip server-ohne --tcp=80:8080
kubectl -n tag17 create service clusterip server-mit --tcp=80:8080
kubectl -n tag17 rollout status deploy/last-mit --timeout=180s
kubectl -n tag17 rollout status deploy/server-mit --timeout=180s
kubectl -n tag17 get pods
kubectl -n tag17 get pods \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,QOS:.status.qosClass

Expected:

NAME                          READY   STATUS    RESTARTS   AGE
last-mit-5d49799bd4-sgk2q     2/2     Running   0          8s
last-ohne-6d98856885-pb5h4    1/1     Running   0          8s
server-mit-66ddd9d74d-fnzrn   2/2     Running   0          8s
server-ohne-fcfff95d7-5g6lh   1/1     Running   0          8s
NAME                          NODE      QOS
last-mit-5d49799bd4-sgk2q     worker1   BestEffort
last-ohne-6d98856885-pb5h4    worker1   BestEffort
server-mit-66ddd9d74d-fnzrn   worker2   BestEffort
server-ohne-fcfff95d7-5g6lh   worker2   BestEffort

fortio is a load generator with a fixed rate and a latency distribution; the image is available for amd64 and arm64. fortio server only keeps the Pod alive; you start the measurement via exec. Remember the QOS column: no Pod, not even the proxies, has requests.

Save your progress? Sign up for free

Drill 2: An idle proxy

sleep 40
kubectl -n tag17 top pod --containers
linkerd diagnostics proxy-metrics -n tag17 deploy/server-mit \
  | grep -E '^process_(resident_memory|threads)'
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep runtime

Expected (abridged):

POD                           NAME            CPU(cores)   MEMORY(bytes)
last-mit-5d49799bd4-sgk2q     fortio          0m           4Mi
last-mit-5d49799bd4-sgk2q     linkerd-proxy   2m           3Mi
last-ohne-6d98856885-pb5h4    fortio          0m           4Mi
server-mit-66ddd9d74d-fnzrn   httpbin         0m           2Mi
server-mit-66ddd9d74d-fnzrn   linkerd-proxy   2m           3Mi
server-ohne-fcfff95d7-5g6lh   httpbin         0m           1Mi
process_resident_memory_bytes 22282240
process_threads 2
...  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime

Two numbers for the same memory: kubectl top shows the working set of the container cgroup (3 Mi), the proxy itself reports its RSS (22 MB). The difference is mostly pages of the proxy binary, which all proxies on a node share. For limits and eviction, the working set counts; for cautious planning, take the RSS as the upper bound. And: by default, the proxy works with one worker thread.

Save your progress? Sign up for free

Drill 3: Latency at 100 requests per second

Same load on both setups: 100 QPS, 60 s, 4 connections. With -json -, fortio writes the result as JSON to stdout; logs go to stderr.

kubectl -n tag17 exec deploy/last-ohne -- fortio load -qps 100 -t 60s -c 4 \
  -p 50,90,99 -json - http://server-ohne/get 2>/dev/null > ohne-100.json
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 100 -t 60s -c 4 \
  -p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-100.json
for f in ohne-100.json mit-100.json; do
  jq -c '[(.ActualQPS|round),
  (.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' $f
done
jq -c .RetCodes mit-100.json

Expected (QPS, p50, p90, p99 in ms; numbers vary):

[100,4.8,21.4,116]
[100,13.3,35.8,96.2]
{"200":6000}

Without a proxy, the median is around 4 ms; with a proxy, around 13 ms. The p99 of the variant without a proxy is even worse here than with one. A single run says little. Without -json, fortio writes text output to stderr, with several # target blocks (sleep times, response times, connection setup) – easy to mix up.

Save your progress? Sign up for free

Drill 4: 500 requests per second and CPU under load

During the measurement, you look at the CPU after 45 s. kubectl top averages over a window, so do not measure at the start of the run.

kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 500 -t 60s -c 4 \
  -p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-500.json &
sleep 45; kubectl -n tag17 top pod --containers; wait
kubectl -n tag17 exec deploy/last-ohne -- fortio load -qps 500 -t 60s -c 4 \
  -p 50,90,99 -json - http://server-ohne/get 2>/dev/null > ohne-500.json
for f in ohne-500.json mit-500.json; do
  jq -c '[(.ActualQPS|round),
  (.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' $f
done

Expected (abridged, numbers vary):

POD                           NAME            CPU(cores)   MEMORY(bytes)
last-mit-5d49799bd4-sgk2q     fortio          86m          11Mi
last-mit-5d49799bd4-sgk2q     linkerd-proxy   232m         3Mi
server-mit-66ddd9d74d-fnzrn   httpbin         204m         7Mi
server-mit-66ddd9d74d-fnzrn   linkerd-proxy   222m         3Mi
[500,3.2,10,36.9]
[286,10,26.2,73.7]

The variant with a proxy does not reach 500 QPS, only 286. fortio works closed-loop with 4 connections: each one only sends the next request once the response has arrived. At around 14 ms on average, that is at most 4 / 0.014 s ≈ 285 requests per second. Each proxy needs at least as much CPU for this as the app itself; memory stays at 3 Mi.

Save your progress? Sign up for free

Drill 5: Several runs, median, variation

The workers share their CPU with everything else in the lab (Cilium, Prometheus, the notes app, other namespaces). So repeat Drills 3 and 4 three times and record the values. This is what it looked like in the lab (median of three runs, range in parentheses):

100 QPS p50 (ms) p90 (ms) p99 (ms)
without proxy 3.7 (3.6–3.9) 11.7 (11.2–12.2) 45 (45–56)
with proxy 13.3 (13.3–13.8) 34.3 (34.1–39.6) 95 (92–120)
500 QPS achieved p50 (ms) p99 (ms)
without proxy 500 3.3 (3.3–3.3) 36 (33–39)
with proxy 277 (271–297) 10.2 (9.8–10.5) 74 (66–76)

Around +10 ms in the median per request from two proxies – considerably more than the “under one millisecond” you often read. That is not down to the proxy alone, but to how it runs here. You clear that up in Drill 7.

Save your progress? Sign up for free

Drill 6: Control Plane and extrapolation

kubectl top pod -n linkerd --containers
kubectl top pod -n linkerd --containers --no-headers \
  | awk '{c+=$3; m+=$4} END {print c"m", m"Mi"}'
kubectl top pod -n linkerd-viz --no-headers \
  | awk '{c+=$2; m+=$3} END {print c"m", m"Mi"}'
kubectl top pod -A --containers --no-headers \
  | awk '$3=="linkerd-proxy" {n++; c+=$4; m+=$5} END {print n, c"m", m"Mi"}'

Expected (abridged):

POD                                      NAME             CPU(cores)   MEMORY(bytes)
linkerd-destination-794fdcc84c-8nsx8     destination      5m           25Mi
linkerd-destination-794fdcc84c-8nsx8     linkerd-proxy    7m           8Mi
linkerd-destination-794fdcc84c-8nsx8     policy           2m           5Mi
linkerd-destination-794fdcc84c-8nsx8     sp-validator     1m           10Mi
linkerd-identity-7f569dc484-kng2s        identity         1m           11Mi
...
linkerd-proxy-injector-98fbd8b8c-gdvnj   proxy-injector   2m           21Mi
24m 86Mi
16m 85Mi
24 105m 79Mi

The Control Plane needs just under 90 Mi, viz about the same again. 24 proxies run in the whole cluster. Extrapolated to 100 Pods with the measurements from Drills 2 and 4:

Item per proxy 100 Pods
CPU idle 2 m 0.2 CPU
Memory (working set) 3 Mi 300 Mi
Memory (RSS, upper bound) 22 MB 2.2 GB
CPU under load ~0.8 m per request/s 1000 req/s: ~1.6 CPU

The last line counts twice because every request goes through two proxies. The Control Plane does not grow per Pod, but with the number of endpoints and watches.

Save your progress? Sign up for free

Drill 7: Proxy resources via annotation

The workers are busy, and all Pods are BestEffort: under CPU pressure they only get leftovers. Give the two proxies requests:

R='{"spec":{"template":{"metadata":{"annotations":{
  "config.linkerd.io/proxy-cpu-request":"200m",
  "config.linkerd.io/proxy-memory-request":"20Mi"}}}}}'
for d in last-mit server-mit; do kubectl -n tag17 patch deploy $d -p "$R"; done
kubectl -n tag17 rollout status deploy/last-mit --timeout=120s
kubectl -n tag17 rollout status deploy/server-mit --timeout=120s
sleep 30
kubectl -n tag17 get pod -l app=server-mit -o \
  jsonpath='{.items[0].spec.initContainers[?(@.name=="linkerd-proxy")].resources}{"\n"}'
kubectl -n tag17 get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass

Expected:

{"requests":{"cpu":"200m","memory":"20Mi"}}
NAME                          QOS
last-mit-6b7695dc48-dth4k     Burstable
last-ohne-6d98856885-pb5h4    BestEffort
server-mit-76bc45d6c8-h6jd5   Burstable
server-ohne-fcfff95d7-5g6lh   BestEffort

The sleep 30 is necessary: right after the rollout, items[0] often still returns the old, terminating Pod. Now the same measurements:

kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 100 -t 60s -c 4 \
  -p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-100-req.json
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 500 -t 60s -c 4 \
  -p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-500-req.json
for f in mit-100-req.json mit-500-req.json; do
  jq -c '[(.ActualQPS|round),
  (.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' $f
done

Expected (numbers vary):

[100,8.6,14.8,31.7]
[486,7,13.5,24.7]

In the lab over three runs (median, range):

with requests achieved p50 (ms) p99 (ms)
100 QPS 100 8.7 (8.5–8.7) 27 (27–38)
500 QPS 487 (481–499) 7.0 (6.7–7.0) 26 (25–26)

The overhead in the median halves to around +5 ms, the p99 drops from 95 to 27 ms, and 500 QPS almost get through. However, the comparison is no longer fair: the request raises the CPU weight of the whole Pod, so fortio and httpbin benefit too. The -ohne Pods stay BestEffort. Lesson: here, requests have a stronger effect than the proxy.

Save your progress? Sign up for free

Drill 8: Limits and worker threads

A CPU limit throttles the proxy hard. In the lab, the server proxy needed around 90 m at 100 QPS (measured as in Drill 4) – give it 50 m:

L='{"spec":{"template":{"metadata":{"annotations":{
  "config.linkerd.io/proxy-cpu-request":"50m",
  "config.linkerd.io/proxy-cpu-limit":"50m"}}}}}'
kubectl -n tag17 patch deploy server-mit -p "$L"
kubectl -n tag17 rollout status deploy/server-mit --timeout=120s
sleep 30
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 100 -t 60s -c 4 \
  -p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-100-limit.json
jq -c '[(.ActualQPS|round),
  (.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' mit-100-limit.json

Expected (numbers vary):

[59,91.6,106.7,196.5]

Median over 90 ms, only 59 QPS left. A limit that is too tight is more expensive than any mesh.

The CPU limit also controls the worker threads. The proxy injector converts the limit into LINKERD2_PROXY_CORES_MAX (rounded up); without a limit, it stays at one thread. Try two cores:

Z='{"spec":{"template":{"metadata":{"annotations":{
  "config.linkerd.io/proxy-cpu-request":null,
  "config.linkerd.io/proxy-cpu-limit":"2"}}}}}'
kubectl -n tag17 patch deploy server-mit -p "$Z"
sleep 20
kubectl -n tag17 get pods -l app=server-mit
kubectl -n tag17 get events --field-selector reason=FailedScheduling \
  -o custom-columns=MSG:.message --no-headers | tail -1
kubectl -n tag17 get pod -l app=server-mit --field-selector=status.phase=Pending -o \
  jsonpath='{.items[0].spec.initContainers[?(@.name=="linkerd-proxy")].resources}{"\n"}'

Expected (abridged):

NAME                          READY   STATUS    RESTARTS   AGE
server-mit-788b76dfcc-27tf7   0/2     Pending   0          21s
server-mit-7dcdd7dc98-lbfx5   2/2     Running   0          2m21s
0/3 nodes are available: 1 Insufficient cpu, 1 node(s) didn't match Pod's node ...
{"limits":{"cpu":"2"},"requests":{"cpu":"2","memory":"20Mi"}}

Without a CPU request, Kubernetes sets the request equal to the limit: 2 CPU, more than a worker has. The old Pod keeps running, the rollout hangs. Request and limit set separately:

F='{"spec":{"template":{"metadata":{"annotations":{
  "config.linkerd.io/proxy-cpu-request":"200m",
  "config.linkerd.io/proxy-cpu-limit":"2"}}}}}'
kubectl -n tag17 patch deploy server-mit -p "$F"
kubectl -n tag17 rollout status deploy/server-mit --timeout=120s
sleep 30
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep runtime
P=$(kubectl -n tag17 get pod -l app=server-mit -o name)
kubectl -n tag17 get $P -o \
  jsonpath='{.spec.initContainers[?(@.name=="linkerd-proxy")].env}' \
  | jq -c '.[] | select(.name | test("CORES"))'

Expected (abridged):

...  INFO ThreadId(01) linkerd2_proxy::rt: Using multi-threaded proxy runtime cores=2
{"name":"LINKERD2_PROXY_CORES","value":"1"}
{"name":"LINKERD2_PROXY_CORES_MIN","value":"1"}
{"name":"LINKERD2_PROXY_CORES_MAX","value":"2"}

The log says cores=2, the environment CORES=1 – no contradiction. CORES_MAX comes from the limit, CORES_MIN from the request (both rounded up, at least 1). LINKERD2_PROXY_CORES is the old fixed setting (marked deprecated in the chart); the injector sets it equal to CORES_MIN. The proxy starts as many threads as CORES_MAX and the node’s CPUs allow. Cross-checked, each with a request of 200m:

Limit CORES_MAX Proxy log
1500m 2 multi-threaded proxy runtime cores=2
500m 1 single-threaded proxy runtime
4 4 cores=2 – the workers only have two vCPUs

A request of 1500m raises CORES and CORES_MIN to 2.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the QoS class of all Pods in tag17. Target time: 30 s
  2. Show CPU and memory of all proxies in notes. Target time: 30 s
  3. Show how many threads the proxy of server-mit runs with. Target time: 20 s
  4. Measure only the p99 of server-mit in ms over 10 s at 50 QPS. Target time: 60 s

Solution

kubectl -n tag17 get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
kubectl top pod -n notes --containers | grep -E 'NAME|linkerd-proxy'
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep runtime
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 50 -t 10s -p 99 \
  -json - http://server-mit/get 2>/dev/null \
  | jq '.DurationHistogram.Percentiles[0].Value*1000'

Save your progress? Sign up for free

Checkpoint

kubectl -n tag17 get pods

Expected: four Pods Running, the -mit Pods 2/2.

kubectl -n tag17 get pod -l app=server-mit -o jsonpath='{.items[0].status.qosClass}'

Expected: Burstable

kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep -c multi-threaded

Expected: 1

jq -c .RetCodes mit-500-req.json

Expected: only "200", no other code.

Challenge

Give the proxy of last-mit two worker threads as well, without the Pod getting stuck, and measure 500 QPS three times. Does the second thread bring anything compared to Drill 7?

Hint: As in Drill 8 – set request and limit separately.

Solution

F='{"spec":{"template":{"metadata":{"annotations":{
  "config.linkerd.io/proxy-cpu-request":"200m",
  "config.linkerd.io/proxy-cpu-limit":"2"}}}}}'
kubectl -n tag17 patch deploy last-mit -p "$F"
kubectl -n tag17 rollout status deploy/last-mit --timeout=120s
sleep 30
kubectl -n tag17 logs deploy/last-mit -c linkerd-proxy | grep runtime
for i in 1 2 3; do
  kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 500 -t 60s -c 4 \
    -p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-500-2t-$i.json
done
for i in 1 2 3; do
  jq -c '[(.ActualQPS|round),
  (.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' mit-500-2t-$i.json
done

Expected (abridged, numbers vary):

...  INFO ThreadId(01) linkerd2_proxy::rt: Using multi-threaded proxy runtime cores=2
[458,7.3,14.3,28]
[440,7.7,15,29.2]
[463,7.3,14.3,27.4]

No gain, if anything slightly worse than with one thread (Drill 7: 487 QPS, p99 26 ms). One thread handles 500 QPS with around 0.4 CPU. On workers with 2 vCPU that are busy anyway, there is no free core for a second thread. More threads only pay off for proxies with very high throughput, for example at ingress or gateways.

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag17

Interview Note

  • What does a service mesh cost? Two proxy hops per request (latency, CPU on both sides), memory per Pod and a Control Plane. Numbers only from your own measurement at a fixed rate, compared against no proxy.
  • Why is the Linkerd proxy lean? Rust, no garbage collector, a few MB of working set when idle; one worker thread by default, more via the CPU limit.
  • Why set requests for the proxy? Without them, it is BestEffort and loses first on full nodes – latency rises sharply.

Enterprise Note

  • Set proxy requests cluster-wide via the Helm values proxy.resources.*, and deviating workloads via annotation on the Pod or namespace.
  • Use CPU limits for the proxy sparingly or not at all: throttling hits every request of the Pod. If you set a limit, make it generous and always pair it with its own request.
  • Capacity planning: proxy CPU grows with requests per second, not with the number of Pods. Measure high-throughput gateways and ingress individually and allow more threads.
  • Reproduce production in load tests: real payloads, HTTP/2 and gRPC behave differently than /get over HTTP/1.1, and mTLS handshakes weigh more heavily with short-lived connections.
  • Dashboards for proxy CPU and container_cpu_cfs_throttled_periods_total per Pod show where a proxy is being throttled.