Day 0: The Training Ground
Goal: A cluster with a small app made of four services and a load generator – still without a mesh, so you can measure every difference later. · Duration: approx. 45 min · Prerequisite: Kubernetes cluster with three nodes, kubectl and Helm
Briefing
- Linkerd only shows its value with real traffic between several
services. That is why today you build the app
notes, which you practice on throughout the whole book. web(nginx) is the entry point. It forwards/api/tonotes-api,/flakyand/slowtohelper.helperdeliberately fails 30 percent of calls.notes-apistores data inpostgres.lastcallswebwithout pause and writes every status code to its log – that is your measuring instrument.- Today everything runs without a mesh. Today’s numbers are the baseline for Days 1 to 17.
last ──> web ──/api/──> notes-api ──> postgres
│
└──/flaky, /slow──> helper (30 % errors, 2 s delay)
Drill 1: Check the cluster
The book was tested on a kubeadm cluster with three nodes:
cp1 (4 vCPU, 12 GB) and two workers (2 vCPU, 4 GB each).
How to build such a cluster is shown in “CKA hands-on” and “Cilium
hands-on”. All commands run on cp1.
kubectl get nodes
kubectl version | head -2
kubectl get storageclass
helm version --shortExpected:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 12h v1.37.2
worker1 Ready <none> 12h v1.37.2
worker2 Ready <none> 12h v1.37.2
Client Version: v1.37.2
Kustomize Version: v5.8.1
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ...
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer ...
v4.3.2+g8346aaa
Without a StorageClass, Postgres will later stay in
Pending. In that case, install Rancher’s
local-path-provisioner.
Save your progress? Sign up for free
Drill 2: Check Cilium for Linkerd
If Cilium replaces kube-proxy, it can rewrite Service addresses already in the socket. Then the Linkerd proxy only sees Pod IPs and no longer knows which Service the app meant. Retries, routes and traffic splitting then do not take effect.
kubectl -n kube-system get configmap cilium-config \
-o jsonpath='{.data.bpf-lb-sock} {.data.bpf-lb-sock-hostns-only}{"\n"}'Expected:
false true
false means: socket load balancing is off, everything is
fine. If it says true true, everything is fine as well.
Only with true false do you switch:
helm upgrade cilium cilium/cilium -n kube-system --reuse-values \
--set socketLB.hostNamespaceOnly=true
kubectl -n kube-system rollout restart daemonset/ciliumIf you use a different CNI such as Calico or Flannel, skip this drill.
Save your progress? Sign up for free
Drill 3: Namespace and Secrets
mkdir -p ~/linkerd/tag00 && cd ~/linkerd/tag00
kubectl create namespace notes
kubectl -n notes create secret generic notes-db \
--from-literal=password=lab-db-123 \
--from-literal=url='postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable'
kubectl -n notes create secret generic notes-api-key --from-literal=key=lab-key-123Expected:
namespace/notes created
secret/notes-db created
secret/notes-api-key created
The values are lab values. sslmode=disable is
intentional here: from Day 3 on, Linkerd encrypts the connection without
the app knowing anything about it.
Save your progress? Sign up for free
Drill 4: Postgres
cat > postgres.yaml <<'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: postgres, namespace: notes}
spec:
serviceName: postgres
replicas: 1
selector: {matchLabels: {app: postgres}}
template:
metadata: {labels: {app: postgres}}
spec:
containers:
- name: postgres
image: docker.io/library/postgres:18.7-alpine
env:
- {name: POSTGRES_USER, value: notes}
- {name: POSTGRES_DB, value: notes}
- name: POSTGRES_PASSWORD
valueFrom: {secretKeyRef: {name: notes-db, key: password}}
- {name: PGDATA, value: /var/lib/postgresql/data/pgdata}
ports: [{containerPort: 5432, name: postgres}]
readinessProbe:
exec: {command: [pg_isready, -U, notes, -d, notes]}
volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
volumeClaimTemplates:
- metadata: {name: data}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Service
metadata: {name: postgres, namespace: notes}
spec:
selector: {app: postgres}
ports: [{port: 5432, targetPort: postgres, name: postgres}]
EOF
kubectl apply -f postgres.yaml
kubectl -n notes rollout status statefulset/postgres --timeout=180sExpected:
statefulset.apps/postgres created
service/postgres created
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...
Save your progress? Sign up for free
Drill 5: notes-api
The init container migrate creates the table before the
app starts. Remember it – you will run into it on Day 2.
cat > notes-api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: notes}
spec:
replicas: 2
selector: {matchLabels: {app: notes-api}}
template:
metadata: {labels: {app: notes-api}}
spec:
initContainers:
- name: migrate
image: ghcr.io/thomaszachmann/notes-api:0.7.1
args: [migrate]
env:
- name: NOTES_DB_URL
valueFrom: {secretKeyRef: {name: notes-db, key: url}}
containers:
- name: notes-api
image: ghcr.io/thomaszachmann/notes-api:0.7.1
ports: [{containerPort: 8080, name: http}]
env:
- name: NOTES_DB_URL
valueFrom: {secretKeyRef: {name: notes-db, key: url}}
- name: NOTES_API_KEY
valueFrom: {secretKeyRef: {name: notes-api-key, key: key}}
readinessProbe:
httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: notes}
spec:
selector: {app: notes-api}
ports: [{port: 80, targetPort: http, name: http}]
EOF
kubectl apply -f notes-api.yaml
kubectl -n notes rollout status deploy/notes-api --timeout=180s
kubectl -n notes logs deploy/notes-api -c migrateExpected (abridged):
deployment "notes-api" successfully rolled out
Found 2 pods, using pod/notes-api-6d87674cb5-z7kc6
time=2026-11-19T11:16:01.348Z level=INFO msg="migration complete" table=notes
Save your progress? Sign up for free
Drill 6: helper and web
helper is go-httpbin.
/unstable?failure_rate=0.3 responds with 500 in 30 percent
of cases, /delay/2 waits two seconds. web sets
the API key for notes-api.
cat > helper.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: notes}
spec:
replicas: 2
selector: {matchLabels: {app: helper}}
template:
metadata: {labels: {app: helper}}
spec:
containers:
- name: helper
image: ghcr.io/mccutchen/go-httpbin:2.26.0
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: notes}
spec:
selector: {app: helper}
ports: [{port: 80, targetPort: http, name: http}]
EOF
cat > web.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: notes}
data:
default.conf: |
server {
listen 8080;
location = / { default_type text/plain; return 200 "notes web\n"; }
location /api/ {
proxy_set_header X-API-Key lab-key-123;
proxy_pass http://notes-api.notes.svc.cluster.local/;
}
location /flaky {
proxy_pass http://helper.notes.svc.cluster.local/unstable?failure_rate=0.3;
}
location /slow {
proxy_pass http://helper.notes.svc.cluster.local/delay/2;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: notes}
spec:
replicas: 2
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
containers:
- name: web
image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
ports: [{containerPort: 8080, name: http}]
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: notes}
spec:
selector: {app: web}
ports: [{port: 80, targetPort: http, name: http}]
EOF
kubectl apply -f helper.yaml -f web.yaml
kubectl -n notes rollout status deploy/helper --timeout=180s
kubectl -n notes rollout status deploy/web --timeout=180sExpected (abridged):
deployment.apps/helper created
service/helper created
configmap/web-conf created
deployment.apps/web created
service/web created
deployment "helper" successfully rolled out
deployment "web" successfully rolled out
Test the path once by hand, with a throwaway Pod:
kubectl -n notes run t --rm -i -q --image=docker.io/curlimages/curl:8.23.0 \
--restart=Never -- sh -c 'curl -s -X POST -H "Content-Type: application/json" \
-d "{\"text\":\"Tag 0\"}" http://web/api/notes; echo; \
curl -s -o /dev/null -w "%{time_total}s\n" http://web/slow'Expected:
{"id":"3eb62b56-978a-4899-8540-b971f6a1ef87","text":"Tag 0"}
2.012089s
Save your progress? Sign up for free
Drill 7: Load generator and baseline
cat > last.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: notes}
spec:
replicas: 1
selector: {matchLabels: {app: last}}
template:
metadata: {labels: {app: last}}
spec:
containers:
- name: last
image: docker.io/curlimages/curl:8.23.0
command: [sh, -c]
args:
- |
while true; do
for p in /api/notes /api/notes /flaky; do
code=$(curl -s -o /dev/null -w '%{http_code}' -m 5 http://web$p)
echo "$(date +%T) $p $code"
done
sleep 1
done
EOF
kubectl apply -f last.yaml
kubectl -n notes rollout status deploy/lastExpected (abridged):
deployment "last" successfully rolled out
Wait a minute. Then count the status codes per path:
kubectl -n notes logs deploy/last --since=60s | awk '{print $2, $3}' | sort | uniq -cExpected (numbers vary):
92 /api/notes 200
34 /flaky 200
12 /flaky 500
/api/notes always succeeds, /flaky in about
70 percent of cases. Write down both numbers: this is the baseline
without a mesh.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all Pods in
noteswith the node they run on. Target time: 15 s - Count the notes directly in Postgres. Target time: 45 s
- Show the last five lines of the load generator. Target time: 15 s
- Show which Pod IPs the Service
helperpoints to. Target time: 20 s
Solution
kubectl -n notes get pods -o wide
kubectl -n notes exec postgres-0 -- \
psql -U notes -d notes -c 'select count(*) from notes'
kubectl -n notes logs deploy/last --tail=5
kubectl -n notes get endpointslices -l kubernetes.io/service-name=helperSave your progress? Sign up for free
Checkpoint
kubectl -n notes get deployExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
helper 2/2 2 2 5m
last 1/1 1 1 4m
notes-api 2/2 2 2 6m
web 2/2 2 2 5m
kubectl -n notes get statefulset postgresExpected:
NAME READY AGE
postgres 1/1 7m
kubectl -n notes logs deploy/last --tail=30 | grep -c ' 200$'Expected: a number around 27 – about nine out of ten calls succeed.
Challenge
Set the error rate of /flaky to 50 percent and measure
for one minute. Then restore 30 percent.
Hint: nginx only reads the ConfigMap at startup.
Solution
kubectl -n notes get configmap web-conf -o yaml \
| sed 's/failure_rate=0.3/failure_rate=0.5/' | kubectl apply -f -
kubectl -n notes rollout restart deploy/web
kubectl -n notes rollout status deploy/web
sleep 60
kubectl -n notes logs deploy/last --since=50s | awk '$2=="/flaky" {print $3}' \
| sort | uniq -cExpected:
22 200
21 500
Back to 30 percent:
kubectl apply -f ~/linkerd/tag00/web.yaml
kubectl -n notes rollout restart deploy/webThen look at the last lines of last. During the restart,
a 000 often shows up: a connection went nowhere because an
old nginx Pod had already terminated. You will measure this again later
with the mesh.
Save your progress? Sign up for free
Interview Note
- Why measure without a mesh first? Without a baseline, you cannot prove what the mesh brings or costs – neither for errors nor for latency.
- What is a service mesh? A proxy next to every Pod that encrypts, measures and controls connections, plus a Control Plane that configures these proxies.
Enterprise Note
- A test app with deliberate errors belongs in every mesh rollout – with real services, you cannot see whether a retry helped.
- Load generators in production: k6, Fortio or vegeta with fixed rates instead of a curl loop.
- Check the CNI before the rollout. Socket load balancing, custom network policies and CNI chaining determine how Linkerd is installed.