Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: A cluster with a small app made of four services and a load generator – still without a mesh, so you can measure every difference later. · Duration: approx. 45 min · Prerequisite: Kubernetes cluster with three nodes, kubectl and Helm

Briefing

  • Linkerd only shows its value with real traffic between several services. That is why today you build the app notes, which you practice on throughout the whole book.
  • web (nginx) is the entry point. It forwards /api/ to notes-api, /flaky and /slow to helper. helper deliberately fails 30 percent of calls.
  • notes-api stores data in postgres. last calls web without pause and writes every status code to its log – that is your measuring instrument.
  • Today everything runs without a mesh. Today’s numbers are the baseline for Days 1 to 17.
 last ──> web ──/api/──> notes-api ──> postgres
           │
           └──/flaky, /slow──> helper (30 % errors, 2 s delay)

Drill 1: Check the cluster

The book was tested on a kubeadm cluster with three nodes: cp1 (4 vCPU, 12 GB) and two workers (2 vCPU, 4 GB each). How to build such a cluster is shown in “CKA hands-on” and “Cilium hands-on”. All commands run on cp1.

kubectl get nodes
kubectl version | head -2
kubectl get storageclass
helm version --short

Expected:

NAME      STATUS   ROLES           AGE   VERSION
cp1       Ready    control-plane   12h   v1.37.2
worker1   Ready    <none>          12h   v1.37.2
worker2   Ready    <none>          12h   v1.37.2
Client Version: v1.37.2
Kustomize Version: v5.8.1
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE  ...
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer ...
v4.3.2+g8346aaa

Without a StorageClass, Postgres will later stay in Pending. In that case, install Rancher’s local-path-provisioner.

Save your progress? Sign up for free

Drill 2: Check Cilium for Linkerd

If Cilium replaces kube-proxy, it can rewrite Service addresses already in the socket. Then the Linkerd proxy only sees Pod IPs and no longer knows which Service the app meant. Retries, routes and traffic splitting then do not take effect.

kubectl -n kube-system get configmap cilium-config \
  -o jsonpath='{.data.bpf-lb-sock} {.data.bpf-lb-sock-hostns-only}{"\n"}'

Expected:

false true

false means: socket load balancing is off, everything is fine. If it says true true, everything is fine as well. Only with true false do you switch:

helm upgrade cilium cilium/cilium -n kube-system --reuse-values \
  --set socketLB.hostNamespaceOnly=true
kubectl -n kube-system rollout restart daemonset/cilium

If you use a different CNI such as Calico or Flannel, skip this drill.

Save your progress? Sign up for free

Drill 3: Namespace and Secrets

mkdir -p ~/linkerd/tag00 && cd ~/linkerd/tag00
kubectl create namespace notes
kubectl -n notes create secret generic notes-db \
  --from-literal=password=lab-db-123 \
  --from-literal=url='postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable'
kubectl -n notes create secret generic notes-api-key --from-literal=key=lab-key-123

Expected:

namespace/notes created
secret/notes-db created
secret/notes-api-key created

The values are lab values. sslmode=disable is intentional here: from Day 3 on, Linkerd encrypts the connection without the app knowing anything about it.

Save your progress? Sign up for free

Drill 4: Postgres

cat > postgres.yaml <<'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: postgres, namespace: notes}
spec:
  serviceName: postgres
  replicas: 1
  selector: {matchLabels: {app: postgres}}
  template:
    metadata: {labels: {app: postgres}}
    spec:
      containers:
      - name: postgres
        image: docker.io/library/postgres:18.7-alpine
        env:
        - {name: POSTGRES_USER, value: notes}
        - {name: POSTGRES_DB, value: notes}
        - name: POSTGRES_PASSWORD
          valueFrom: {secretKeyRef: {name: notes-db, key: password}}
        - {name: PGDATA, value: /var/lib/postgresql/data/pgdata}
        ports: [{containerPort: 5432, name: postgres}]
        readinessProbe:
          exec: {command: [pg_isready, -U, notes, -d, notes]}
        volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
  volumeClaimTemplates:
  - metadata: {name: data}
    spec:
      accessModes: [ReadWriteOnce]
      resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Service
metadata: {name: postgres, namespace: notes}
spec:
  selector: {app: postgres}
  ports: [{port: 5432, targetPort: postgres, name: postgres}]
EOF
kubectl apply -f postgres.yaml
kubectl -n notes rollout status statefulset/postgres --timeout=180s

Expected:

statefulset.apps/postgres created
service/postgres created
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...

Save your progress? Sign up for free

Drill 5: notes-api

The init container migrate creates the table before the app starts. Remember it – you will run into it on Day 2.

cat > notes-api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: notes}
spec:
  replicas: 2
  selector: {matchLabels: {app: notes-api}}
  template:
    metadata: {labels: {app: notes-api}}
    spec:
      initContainers:
      - name: migrate
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        args: [migrate]
        env:
        - name: NOTES_DB_URL
          valueFrom: {secretKeyRef: {name: notes-db, key: url}}
      containers:
      - name: notes-api
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        ports: [{containerPort: 8080, name: http}]
        env:
        - name: NOTES_DB_URL
          valueFrom: {secretKeyRef: {name: notes-db, key: url}}
        - name: NOTES_API_KEY
          valueFrom: {secretKeyRef: {name: notes-api-key, key: key}}
        readinessProbe:
          httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: notes}
spec:
  selector: {app: notes-api}
  ports: [{port: 80, targetPort: http, name: http}]
EOF
kubectl apply -f notes-api.yaml
kubectl -n notes rollout status deploy/notes-api --timeout=180s
kubectl -n notes logs deploy/notes-api -c migrate

Expected (abridged):

deployment "notes-api" successfully rolled out
Found 2 pods, using pod/notes-api-6d87674cb5-z7kc6
time=2026-11-19T11:16:01.348Z level=INFO msg="migration complete" table=notes

Save your progress? Sign up for free

Drill 6: helper and web

helper is go-httpbin. /unstable?failure_rate=0.3 responds with 500 in 30 percent of cases, /delay/2 waits two seconds. web sets the API key for notes-api.

cat > helper.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: notes}
spec:
  replicas: 2
  selector: {matchLabels: {app: helper}}
  template:
    metadata: {labels: {app: helper}}
    spec:
      containers:
      - name: helper
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: notes}
spec:
  selector: {app: helper}
  ports: [{port: 80, targetPort: http, name: http}]
EOF
cat > web.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: notes}
data:
  default.conf: |
    server {
      listen 8080;
      location = / { default_type text/plain; return 200 "notes web\n"; }
      location /api/ {
        proxy_set_header X-API-Key lab-key-123;
        proxy_pass http://notes-api.notes.svc.cluster.local/;
      }
      location /flaky {
        proxy_pass http://helper.notes.svc.cluster.local/unstable?failure_rate=0.3;
      }
      location /slow {
        proxy_pass http://helper.notes.svc.cluster.local/delay/2;
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: notes}
spec:
  replicas: 2
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: notes}
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: http, name: http}]
EOF
kubectl apply -f helper.yaml -f web.yaml
kubectl -n notes rollout status deploy/helper --timeout=180s
kubectl -n notes rollout status deploy/web --timeout=180s

Expected (abridged):

deployment.apps/helper created
service/helper created
configmap/web-conf created
deployment.apps/web created
service/web created
deployment "helper" successfully rolled out
deployment "web" successfully rolled out

Test the path once by hand, with a throwaway Pod:

kubectl -n notes run t --rm -i -q --image=docker.io/curlimages/curl:8.23.0 \
  --restart=Never -- sh -c 'curl -s -X POST -H "Content-Type: application/json" \
  -d "{\"text\":\"Tag 0\"}" http://web/api/notes; echo; \
  curl -s -o /dev/null -w "%{time_total}s\n" http://web/slow'

Expected:

{"id":"3eb62b56-978a-4899-8540-b971f6a1ef87","text":"Tag 0"}
2.012089s

Save your progress? Sign up for free

Drill 7: Load generator and baseline

cat > last.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: notes}
spec:
  replicas: 1
  selector: {matchLabels: {app: last}}
  template:
    metadata: {labels: {app: last}}
    spec:
      containers:
      - name: last
        image: docker.io/curlimages/curl:8.23.0
        command: [sh, -c]
        args:
        - |
          while true; do
            for p in /api/notes /api/notes /flaky; do
              code=$(curl -s -o /dev/null -w '%{http_code}' -m 5 http://web$p)
              echo "$(date +%T) $p $code"
            done
            sleep 1
          done
EOF
kubectl apply -f last.yaml
kubectl -n notes rollout status deploy/last

Expected (abridged):

deployment "last" successfully rolled out

Wait a minute. Then count the status codes per path:

kubectl -n notes logs deploy/last --since=60s | awk '{print $2, $3}' | sort | uniq -c

Expected (numbers vary):

     92 /api/notes 200
     34 /flaky 200
     12 /flaky 500

/api/notes always succeeds, /flaky in about 70 percent of cases. Write down both numbers: this is the baseline without a mesh.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all Pods in notes with the node they run on. Target time: 15 s
  2. Count the notes directly in Postgres. Target time: 45 s
  3. Show the last five lines of the load generator. Target time: 15 s
  4. Show which Pod IPs the Service helper points to. Target time: 20 s

Solution

kubectl -n notes get pods -o wide
kubectl -n notes exec postgres-0 -- \
  psql -U notes -d notes -c 'select count(*) from notes'
kubectl -n notes logs deploy/last --tail=5
kubectl -n notes get endpointslices -l kubernetes.io/service-name=helper

Save your progress? Sign up for free

Checkpoint

kubectl -n notes get deploy

Expected:

NAME        READY   UP-TO-DATE   AVAILABLE   AGE
helper      2/2     2            2           5m
last        1/1     1            1           4m
notes-api   2/2     2            2           6m
web         2/2     2            2           5m
kubectl -n notes get statefulset postgres

Expected:

NAME       READY   AGE
postgres   1/1     7m
kubectl -n notes logs deploy/last --tail=30 | grep -c ' 200$'

Expected: a number around 27 – about nine out of ten calls succeed.

Challenge

Set the error rate of /flaky to 50 percent and measure for one minute. Then restore 30 percent.

Hint: nginx only reads the ConfigMap at startup.

Solution

kubectl -n notes get configmap web-conf -o yaml \
  | sed 's/failure_rate=0.3/failure_rate=0.5/' | kubectl apply -f -
kubectl -n notes rollout restart deploy/web
kubectl -n notes rollout status deploy/web
sleep 60
kubectl -n notes logs deploy/last --since=50s | awk '$2=="/flaky" {print $3}' \
  | sort | uniq -c

Expected:

     22 200
     21 500

Back to 30 percent:

kubectl apply -f ~/linkerd/tag00/web.yaml
kubectl -n notes rollout restart deploy/web

Then look at the last lines of last. During the restart, a 000 often shows up: a connection went nowhere because an old nginx Pod had already terminated. You will measure this again later with the mesh.

Save your progress? Sign up for free

Interview Note

  • Why measure without a mesh first? Without a baseline, you cannot prove what the mesh brings or costs – neither for errors nor for latency.
  • What is a service mesh? A proxy next to every Pod that encrypts, measures and controls connections, plus a Control Plane that configures these proxies.

Enterprise Note

  • A test app with deliberate errors belongs in every mesh rollout – with real services, you cannot see whether a retry helped.
  • Load generators in production: k6, Fortio or vegeta with fixed rates instead of a curl loop.
  • Check the CNI before the rollout. Socket load balancing, custom network policies and CNI chaining determine how Linkerd is installed.