Day 17: What the Mesh Costs
Goal: You measure for yourself how much latency, CPU and memory the proxy costs, and control its resources via annotations. · Duration: approx. 90 min · Prerequisite: Day 4
Briefing
- Every request in the mesh passes through two proxies: the client’s and the server’s. That costs latency and CPU on both sides.
- Do not trust any benchmark, not even one from this book: build the same setup twice, once without and once with a proxy, and measure at a fixed rate.
- Lab measurements fluctuate. Run several times, take the median, state the range.
- You set the proxy’s resources with
config.linkerd.io/proxy-cpu-requestand its relatives. Without them, the proxy is BestEffort.
worker1 worker2
last-ohne (fortio) ───────────────> server-ohne (go-httpbin)
last-mit (fortio) ─> proxy ═mTLS═> proxy ─> server-mit (go-httpbin)
Drill 1: Two identical setups
The namespace stays without injection. Only the two -mit
Deployments get the proxy via a Pod annotation.
nodeSelector keeps clients and servers on fixed nodes;
otherwise you end up comparing nodes instead of proxies.
mkdir -p ~/linkerd/tag17 && cd ~/linkerd/tag17
kubectl create namespace tag17
cat > app.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: server-ohne, namespace: tag17}
spec:
replicas: 1
selector: {matchLabels: {app: server-ohne}}
template:
metadata: {labels: {app: server-ohne}}
spec:
nodeSelector: {kubernetes.io/hostname: worker2}
containers:
- name: httpbin
image: ghcr.io/mccutchen/go-httpbin:2.26.0
ports: [{containerPort: 8080, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: server-mit, namespace: tag17}
spec:
replicas: 1
selector: {matchLabels: {app: server-mit}}
template:
metadata:
labels: {app: server-mit}
annotations: {linkerd.io/inject: enabled}
spec:
nodeSelector: {kubernetes.io/hostname: worker2}
containers:
- name: httpbin
image: ghcr.io/mccutchen/go-httpbin:2.26.0
ports: [{containerPort: 8080, name: http}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last-ohne, namespace: tag17}
spec:
replicas: 1
selector: {matchLabels: {app: last-ohne}}
template:
metadata: {labels: {app: last-ohne}}
spec:
nodeSelector: {kubernetes.io/hostname: worker1}
containers:
- name: fortio
image: docker.io/fortio/fortio:1.69.5
args: [server]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last-mit, namespace: tag17}
spec:
replicas: 1
selector: {matchLabels: {app: last-mit}}
template:
metadata:
labels: {app: last-mit}
annotations: {linkerd.io/inject: enabled}
spec:
nodeSelector: {kubernetes.io/hostname: worker1}
containers:
- name: fortio
image: docker.io/fortio/fortio:1.69.5
args: [server]
EOF
kubectl apply -f app.yaml
kubectl -n tag17 create service clusterip server-ohne --tcp=80:8080
kubectl -n tag17 create service clusterip server-mit --tcp=80:8080
kubectl -n tag17 rollout status deploy/last-mit --timeout=180s
kubectl -n tag17 rollout status deploy/server-mit --timeout=180s
kubectl -n tag17 get pods
kubectl -n tag17 get pods \
-o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,QOS:.status.qosClassExpected:
NAME READY STATUS RESTARTS AGE
last-mit-5d49799bd4-sgk2q 2/2 Running 0 8s
last-ohne-6d98856885-pb5h4 1/1 Running 0 8s
server-mit-66ddd9d74d-fnzrn 2/2 Running 0 8s
server-ohne-fcfff95d7-5g6lh 1/1 Running 0 8s
NAME NODE QOS
last-mit-5d49799bd4-sgk2q worker1 BestEffort
last-ohne-6d98856885-pb5h4 worker1 BestEffort
server-mit-66ddd9d74d-fnzrn worker2 BestEffort
server-ohne-fcfff95d7-5g6lh worker2 BestEffort
fortio is a load generator with a fixed rate and a
latency distribution; the image is available for amd64 and arm64.
fortio server only keeps the Pod alive; you start the
measurement via exec. Remember the QOS column:
no Pod, not even the proxies, has requests.
Save your progress? Sign up for free
Drill 2: An idle proxy
sleep 40
kubectl -n tag17 top pod --containers
linkerd diagnostics proxy-metrics -n tag17 deploy/server-mit \
| grep -E '^process_(resident_memory|threads)'
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep runtimeExpected (abridged):
POD NAME CPU(cores) MEMORY(bytes)
last-mit-5d49799bd4-sgk2q fortio 0m 4Mi
last-mit-5d49799bd4-sgk2q linkerd-proxy 2m 3Mi
last-ohne-6d98856885-pb5h4 fortio 0m 4Mi
server-mit-66ddd9d74d-fnzrn httpbin 0m 2Mi
server-mit-66ddd9d74d-fnzrn linkerd-proxy 2m 3Mi
server-ohne-fcfff95d7-5g6lh httpbin 0m 1Mi
process_resident_memory_bytes 22282240
process_threads 2
... INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
Two numbers for the same memory: kubectl top shows the
working set of the container cgroup (3 Mi), the proxy
itself reports its RSS (22 MB). The difference is
mostly pages of the proxy binary, which all proxies on a node share. For
limits and eviction, the working set counts; for cautious planning, take
the RSS as the upper bound. And: by default, the proxy works with
one worker thread.
Save your progress? Sign up for free
Drill 3: Latency at 100 requests per second
Same load on both setups: 100 QPS, 60 s, 4 connections. With
-json -, fortio writes the result as JSON to stdout; logs
go to stderr.
kubectl -n tag17 exec deploy/last-ohne -- fortio load -qps 100 -t 60s -c 4 \
-p 50,90,99 -json - http://server-ohne/get 2>/dev/null > ohne-100.json
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 100 -t 60s -c 4 \
-p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-100.json
for f in ohne-100.json mit-100.json; do
jq -c '[(.ActualQPS|round),
(.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' $f
done
jq -c .RetCodes mit-100.jsonExpected (QPS, p50, p90, p99 in ms; numbers vary):
[100,4.8,21.4,116]
[100,13.3,35.8,96.2]
{"200":6000}
Without a proxy, the median is around 4 ms; with a proxy, around 13
ms. The p99 of the variant without a proxy is even worse here than with
one. A single run says little. Without -json, fortio writes
text output to stderr, with several
# target blocks (sleep times, response times, connection
setup) – easy to mix up.
Save your progress? Sign up for free
Drill 4: 500 requests per second and CPU under load
During the measurement, you look at the CPU after 45 s.
kubectl top averages over a window, so do not measure at
the start of the run.
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 500 -t 60s -c 4 \
-p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-500.json &
sleep 45; kubectl -n tag17 top pod --containers; wait
kubectl -n tag17 exec deploy/last-ohne -- fortio load -qps 500 -t 60s -c 4 \
-p 50,90,99 -json - http://server-ohne/get 2>/dev/null > ohne-500.json
for f in ohne-500.json mit-500.json; do
jq -c '[(.ActualQPS|round),
(.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' $f
doneExpected (abridged, numbers vary):
POD NAME CPU(cores) MEMORY(bytes)
last-mit-5d49799bd4-sgk2q fortio 86m 11Mi
last-mit-5d49799bd4-sgk2q linkerd-proxy 232m 3Mi
server-mit-66ddd9d74d-fnzrn httpbin 204m 7Mi
server-mit-66ddd9d74d-fnzrn linkerd-proxy 222m 3Mi
[500,3.2,10,36.9]
[286,10,26.2,73.7]
The variant with a proxy does not reach 500 QPS, only 286. fortio works closed-loop with 4 connections: each one only sends the next request once the response has arrived. At around 14 ms on average, that is at most 4 / 0.014 s ≈ 285 requests per second. Each proxy needs at least as much CPU for this as the app itself; memory stays at 3 Mi.
Save your progress? Sign up for free
Drill 5: Several runs, median, variation
The workers share their CPU with everything else in the lab (Cilium, Prometheus, the notes app, other namespaces). So repeat Drills 3 and 4 three times and record the values. This is what it looked like in the lab (median of three runs, range in parentheses):
| 100 QPS | p50 (ms) | p90 (ms) | p99 (ms) |
|---|---|---|---|
| without proxy | 3.7 (3.6–3.9) | 11.7 (11.2–12.2) | 45 (45–56) |
| with proxy | 13.3 (13.3–13.8) | 34.3 (34.1–39.6) | 95 (92–120) |
| 500 QPS | achieved | p50 (ms) | p99 (ms) |
|---|---|---|---|
| without proxy | 500 | 3.3 (3.3–3.3) | 36 (33–39) |
| with proxy | 277 (271–297) | 10.2 (9.8–10.5) | 74 (66–76) |
Around +10 ms in the median per request from two proxies – considerably more than the “under one millisecond” you often read. That is not down to the proxy alone, but to how it runs here. You clear that up in Drill 7.
Save your progress? Sign up for free
Drill 6: Control Plane and extrapolation
kubectl top pod -n linkerd --containers
kubectl top pod -n linkerd --containers --no-headers \
| awk '{c+=$3; m+=$4} END {print c"m", m"Mi"}'
kubectl top pod -n linkerd-viz --no-headers \
| awk '{c+=$2; m+=$3} END {print c"m", m"Mi"}'
kubectl top pod -A --containers --no-headers \
| awk '$3=="linkerd-proxy" {n++; c+=$4; m+=$5} END {print n, c"m", m"Mi"}'Expected (abridged):
POD NAME CPU(cores) MEMORY(bytes)
linkerd-destination-794fdcc84c-8nsx8 destination 5m 25Mi
linkerd-destination-794fdcc84c-8nsx8 linkerd-proxy 7m 8Mi
linkerd-destination-794fdcc84c-8nsx8 policy 2m 5Mi
linkerd-destination-794fdcc84c-8nsx8 sp-validator 1m 10Mi
linkerd-identity-7f569dc484-kng2s identity 1m 11Mi
...
linkerd-proxy-injector-98fbd8b8c-gdvnj proxy-injector 2m 21Mi
24m 86Mi
16m 85Mi
24 105m 79Mi
The Control Plane needs just under 90 Mi, viz about the same again. 24 proxies run in the whole cluster. Extrapolated to 100 Pods with the measurements from Drills 2 and 4:
| Item | per proxy | 100 Pods |
|---|---|---|
| CPU idle | 2 m | 0.2 CPU |
| Memory (working set) | 3 Mi | 300 Mi |
| Memory (RSS, upper bound) | 22 MB | 2.2 GB |
| CPU under load | ~0.8 m per request/s | 1000 req/s: ~1.6 CPU |
The last line counts twice because every request goes through two proxies. The Control Plane does not grow per Pod, but with the number of endpoints and watches.
Save your progress? Sign up for free
Drill 7: Proxy resources via annotation
The workers are busy, and all Pods are BestEffort: under CPU pressure they only get leftovers. Give the two proxies requests:
R='{"spec":{"template":{"metadata":{"annotations":{
"config.linkerd.io/proxy-cpu-request":"200m",
"config.linkerd.io/proxy-memory-request":"20Mi"}}}}}'
for d in last-mit server-mit; do kubectl -n tag17 patch deploy $d -p "$R"; done
kubectl -n tag17 rollout status deploy/last-mit --timeout=120s
kubectl -n tag17 rollout status deploy/server-mit --timeout=120s
sleep 30
kubectl -n tag17 get pod -l app=server-mit -o \
jsonpath='{.items[0].spec.initContainers[?(@.name=="linkerd-proxy")].resources}{"\n"}'
kubectl -n tag17 get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClassExpected:
{"requests":{"cpu":"200m","memory":"20Mi"}}
NAME QOS
last-mit-6b7695dc48-dth4k Burstable
last-ohne-6d98856885-pb5h4 BestEffort
server-mit-76bc45d6c8-h6jd5 Burstable
server-ohne-fcfff95d7-5g6lh BestEffort
The sleep 30 is necessary: right after the rollout,
items[0] often still returns the old, terminating Pod. Now
the same measurements:
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 100 -t 60s -c 4 \
-p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-100-req.json
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 500 -t 60s -c 4 \
-p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-500-req.json
for f in mit-100-req.json mit-500-req.json; do
jq -c '[(.ActualQPS|round),
(.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' $f
doneExpected (numbers vary):
[100,8.6,14.8,31.7]
[486,7,13.5,24.7]
In the lab over three runs (median, range):
| with requests | achieved | p50 (ms) | p99 (ms) |
|---|---|---|---|
| 100 QPS | 100 | 8.7 (8.5–8.7) | 27 (27–38) |
| 500 QPS | 487 (481–499) | 7.0 (6.7–7.0) | 26 (25–26) |
The overhead in the median halves to around +5 ms, the p99 drops from
95 to 27 ms, and 500 QPS almost get through. However, the comparison is
no longer fair: the request raises the CPU weight of the whole
Pod, so fortio and httpbin benefit too. The -ohne
Pods stay BestEffort. Lesson: here, requests have a stronger effect than
the proxy.
Save your progress? Sign up for free
Drill 8: Limits and worker threads
A CPU limit throttles the proxy hard. In the lab, the server proxy needed around 90 m at 100 QPS (measured as in Drill 4) – give it 50 m:
L='{"spec":{"template":{"metadata":{"annotations":{
"config.linkerd.io/proxy-cpu-request":"50m",
"config.linkerd.io/proxy-cpu-limit":"50m"}}}}}'
kubectl -n tag17 patch deploy server-mit -p "$L"
kubectl -n tag17 rollout status deploy/server-mit --timeout=120s
sleep 30
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 100 -t 60s -c 4 \
-p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-100-limit.json
jq -c '[(.ActualQPS|round),
(.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' mit-100-limit.jsonExpected (numbers vary):
[59,91.6,106.7,196.5]
Median over 90 ms, only 59 QPS left. A limit that is too tight is more expensive than any mesh.
The CPU limit also controls the worker threads. The
proxy injector converts the limit into
LINKERD2_PROXY_CORES_MAX (rounded up); without a limit, it
stays at one thread. Try two cores:
Z='{"spec":{"template":{"metadata":{"annotations":{
"config.linkerd.io/proxy-cpu-request":null,
"config.linkerd.io/proxy-cpu-limit":"2"}}}}}'
kubectl -n tag17 patch deploy server-mit -p "$Z"
sleep 20
kubectl -n tag17 get pods -l app=server-mit
kubectl -n tag17 get events --field-selector reason=FailedScheduling \
-o custom-columns=MSG:.message --no-headers | tail -1
kubectl -n tag17 get pod -l app=server-mit --field-selector=status.phase=Pending -o \
jsonpath='{.items[0].spec.initContainers[?(@.name=="linkerd-proxy")].resources}{"\n"}'Expected (abridged):
NAME READY STATUS RESTARTS AGE
server-mit-788b76dfcc-27tf7 0/2 Pending 0 21s
server-mit-7dcdd7dc98-lbfx5 2/2 Running 0 2m21s
0/3 nodes are available: 1 Insufficient cpu, 1 node(s) didn't match Pod's node ...
{"limits":{"cpu":"2"},"requests":{"cpu":"2","memory":"20Mi"}}
Without a CPU request, Kubernetes sets the request equal to the limit: 2 CPU, more than a worker has. The old Pod keeps running, the rollout hangs. Request and limit set separately:
F='{"spec":{"template":{"metadata":{"annotations":{
"config.linkerd.io/proxy-cpu-request":"200m",
"config.linkerd.io/proxy-cpu-limit":"2"}}}}}'
kubectl -n tag17 patch deploy server-mit -p "$F"
kubectl -n tag17 rollout status deploy/server-mit --timeout=120s
sleep 30
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep runtime
P=$(kubectl -n tag17 get pod -l app=server-mit -o name)
kubectl -n tag17 get $P -o \
jsonpath='{.spec.initContainers[?(@.name=="linkerd-proxy")].env}' \
| jq -c '.[] | select(.name | test("CORES"))'Expected (abridged):
... INFO ThreadId(01) linkerd2_proxy::rt: Using multi-threaded proxy runtime cores=2
{"name":"LINKERD2_PROXY_CORES","value":"1"}
{"name":"LINKERD2_PROXY_CORES_MIN","value":"1"}
{"name":"LINKERD2_PROXY_CORES_MAX","value":"2"}
The log says cores=2, the environment
CORES=1 – no contradiction. CORES_MAX comes
from the limit, CORES_MIN from the request (both rounded
up, at least 1). LINKERD2_PROXY_CORES is the old fixed
setting (marked deprecated in the chart); the injector sets it equal to
CORES_MIN. The proxy starts as many threads as
CORES_MAX and the node’s CPUs allow. Cross-checked, each
with a request of 200m:
| Limit | CORES_MAX |
Proxy log |
|---|---|---|
1500m |
2 | multi-threaded proxy runtime cores=2 |
500m |
1 | single-threaded proxy runtime |
4 |
4 | cores=2 – the workers only have two vCPUs |
A request of 1500m raises CORES and
CORES_MIN to 2.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the QoS class of all Pods in
tag17. Target time: 30 s - Show CPU and memory of all proxies in
notes. Target time: 30 s - Show how many threads the proxy of
server-mitruns with. Target time: 20 s - Measure only the p99 of
server-mitin ms over 10 s at 50 QPS. Target time: 60 s
Solution
kubectl -n tag17 get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
kubectl top pod -n notes --containers | grep -E 'NAME|linkerd-proxy'
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep runtime
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 50 -t 10s -p 99 \
-json - http://server-mit/get 2>/dev/null \
| jq '.DurationHistogram.Percentiles[0].Value*1000'Save your progress? Sign up for free
Checkpoint
kubectl -n tag17 get podsExpected: four Pods Running, the
-mit Pods 2/2.
kubectl -n tag17 get pod -l app=server-mit -o jsonpath='{.items[0].status.qosClass}'Expected: Burstable
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep -c multi-threadedExpected: 1
jq -c .RetCodes mit-500-req.jsonExpected: only "200", no other
code.
Challenge
Give the proxy of last-mit two worker threads as well,
without the Pod getting stuck, and measure 500 QPS three times. Does the
second thread bring anything compared to Drill 7?
Hint: As in Drill 8 – set request and limit separately.
Solution
F='{"spec":{"template":{"metadata":{"annotations":{
"config.linkerd.io/proxy-cpu-request":"200m",
"config.linkerd.io/proxy-cpu-limit":"2"}}}}}'
kubectl -n tag17 patch deploy last-mit -p "$F"
kubectl -n tag17 rollout status deploy/last-mit --timeout=120s
sleep 30
kubectl -n tag17 logs deploy/last-mit -c linkerd-proxy | grep runtime
for i in 1 2 3; do
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 500 -t 60s -c 4 \
-p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-500-2t-$i.json
done
for i in 1 2 3; do
jq -c '[(.ActualQPS|round),
(.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' mit-500-2t-$i.json
doneExpected (abridged, numbers vary):
... INFO ThreadId(01) linkerd2_proxy::rt: Using multi-threaded proxy runtime cores=2
[458,7.3,14.3,28]
[440,7.7,15,29.2]
[463,7.3,14.3,27.4]
No gain, if anything slightly worse than with one thread (Drill 7: 487 QPS, p99 26 ms). One thread handles 500 QPS with around 0.4 CPU. On workers with 2 vCPU that are busy anyway, there is no free core for a second thread. More threads only pay off for proxies with very high throughput, for example at ingress or gateways.
Save your progress? Sign up for free
Cleanup
kubectl delete namespace tag17Interview Note
- What does a service mesh cost? Two proxy hops per request (latency, CPU on both sides), memory per Pod and a Control Plane. Numbers only from your own measurement at a fixed rate, compared against no proxy.
- Why is the Linkerd proxy lean? Rust, no garbage collector, a few MB of working set when idle; one worker thread by default, more via the CPU limit.
- Why set requests for the proxy? Without them, it is BestEffort and loses first on full nodes – latency rises sharply.
Enterprise Note
- Set proxy requests cluster-wide via the Helm values
proxy.resources.*, and deviating workloads via annotation on the Pod or namespace. - Use CPU limits for the proxy sparingly or not at all: throttling hits every request of the Pod. If you set a limit, make it generous and always pair it with its own request.
- Capacity planning: proxy CPU grows with requests per second, not with the number of Pods. Measure high-throughput gateways and ingress individually and allow more threads.
- Reproduce production in load tests: real payloads, HTTP/2 and gRPC
behave differently than
/getover HTTP/1.1, and mTLS handshakes weigh more heavily with short-lived connections. - Dashboards for proxy CPU and
container_cpu_cfs_throttled_periods_totalper Pod show where a proxy is being throttled.