Skip to content
Hands-on IT IT Drills

Preface

A service mesh promises a lot: mTLS between all services without touching the app, success rates and latencies for every connection, retries, timeouts, canary releases, identity-based policies, and services that are reachable across cluster boundaries. Whether that holds true, you only find out in operation. Then an init container suddenly bypasses the proxy, a retry triples the load on a service that is already down, the issuer expires at night, or a single wrong ServiceAccount locks out an entire chain with 403. This book prepares you for exactly these moments – not with slides, but in the terminal.

You run Linkerd in a real cluster, on a small app made of four services that makes mistakes on purpose. Every feature starts with a measurement without it and ends with a measurement with it: success rate in the load generator, linkerd viz stat, Prometheus, a capture with tcpdump. What you cannot prove, you have not configured.

The book is aimed at platform and DevOps engineers with Kubernetes basics: you should know Deployments, Services and ServiceAccounts, read YAML, use kubectl and be able to install a Helm chart. You should roughly know how TLS certificates work in principle; everything Linkerd builds on top of that, you learn here. You do not need to have run a service mesh before.

How the book works

  • One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
  • Before, configure, after. Every feature ends with proof: a line in the load generator, a metric from the proxy, a packet on the wire.
  • No helper script. Every step is shown as a single command, including the Checkpoint at the end of each day. You write manifests to files with heredocs.
  • Speed Drills at the end of each day: small tasks with a target time.
  • Checkpoint, Challenge, Interview Note, Enterprise Note close out every day. The Checkpoint uses single commands to check whether your cluster is in the expected state – the following days build on it. The Challenge is a task without instructions, with a hint and a solution. The Interview Note collects the questions that come up in a job interview or on call, along with the core answer. The Enterprise Note tells you what is different in production than in the lab.
  • One app through all days: notes with web (nginx), notes-api (Go), helper (go-httpbin, fails 30 percent of calls) and postgres. The load generator last writes every status code to its log – that is your measuring instrument. Days that break something work in a copy in their own namespace.

To finish:

  • Competition (Day 18): a broken mesh that you repair under time pressure.
  • Appendix A is the cheat sheet, Appendix B the final exam with tasks across the whole book.

The days at a glance

Day Topic Duration
0 Training ground: check the cluster, app notes, baseline without mesh approx. 45 min
1 Install Linkerd: CLI, CRDs, Control Plane, linkerd check approx. 45 min
2 Bring Pods into the mesh: injection, native sidecar, opaque ports approx. 60 min
3 mTLS and identity: proxy certificates, proof with tcpdump approx. 60 min
4 Observe: linkerd-viz, Prometheus, Grafana, tap approx. 75 min
5 Certificates: your own trust anchor, rotation, cert-manager approx. 90 min
6 Authorization: Server, AuthorizationPolicy, default deny approx. 75 min
7 Retries and timeouts with HTTPRoutes approx. 60 min
8 Circuit breaking and rate limiting approx. 60 min
9 Traffic shifting: canary, header routing, rollback approx. 60 min
10 Ingress with Gateway API: Cilium gateway and proxy in the mesh approx. 75 min
11 Egress: EgressNetwork, TLSRoute, HTTPRoute to the outside approx. 60 min
12 Multicluster: second cluster, link, mirrored services approx. 90 min
13 Multicluster: failover and federated services approx. 90 min
14 Linkerd with Helm and highly available approx. 75 min
15 Upgrade and rollback approx. 75 min
16 Troubleshooting: eight typical mesh failures approx. 90 min
17 What the mesh costs: latency, CPU, memory, proxy resources approx. 90 min
18 Competition: the broken mesh approx. 90 min

Time: Days 0 to 16 add up to just under 20 hours (1185 minutes), Day 17 and the Competition add 180 minutes – just under 23 hours in total, plus the final exam. At one training day per evening and five evenings per week, you are through in just under four weeks.

The order is no accident: from Day 6 on, default deny applies in notes, and every later day has to live with it. Day 12 only requires Day 5 (the trust anchor is shared), Days 13 to 15 build on Day 12, Day 16 requires Day 7, Day 17 only Day 4.

The training ground

The book works with two clusters. ost is the main cluster for all days, west joins on Day 12 for multicluster and carries the Helm installation on Days 14 and 15. All commands run as ubuntu on cp1, the working directory is ~/linkerd/tagNN/.

Role Purpose IP in the book In the lab
cp1 Control Plane ost, workstation 172.16.0.126 4 vCPU, 12 GB RAM
worker1 Worker ost 172.16.0.228 2 vCPU, 4 GB RAM
worker2 Worker ost 172.16.0.83 2 vCPU, 4 GB RAM
west k3s single node (from Day 12) 172.16.0.182 4 vCPU, 7 GB RAM
  • ost: kubeadm 1.37.2 with three nodes, Cilium 1.20.5 as the CNI without kube-proxy, Gateway API with the Cilium GatewayClass cilium, a StorageClass local-path and Helm. How to build such a cluster is shown in “CKA hands-on” and “Cilium hands-on”; if you still have the training ground, use it.
  • west: another VM with Ubuntu 24.04, on which Day 12 installs k3s v1.37.2 as a single node. cp1 needs SSH access to west (a user with sudo), and the Pod and Service networks must not overlap with ost.
  • Minimum: The lab values are generous because exercises from other books were also running on the VMs. For this book alone, in our estimation, two workers with 2 vCPU and 4 GB each, a Control Plane with 4 vCPU and 8 GB, and 2 vCPU and 4 GB for west are enough. We have not tested anything smaller.

To be honest about what the days require:

  • Day 0: Cilium’s socket load balancing must not rewrite Service addresses inside the Pod, otherwise routes and traffic shifting do not take effect. Drill 2 checks this. With Calico or Flannel, the drill is skipped.
  • Day 4: you install kube-prometheus-stack yourself (release kps in the namespace monitoring). If you already have one, substitute the release name and namespace.
  • Day 5: cert-manager v1.21.3. If it is not running, the installation command is in the chapter. You also need step and cmctl on cp1, plus jq from Day 4 on.
  • Day 10: a Gateway API controller. In the lab that is Cilium; without LoadBalancer IPs, you access it via NodePort.
  • Day 11: Pods need access to the internet (example.com, linkerd.io).
  • Day 13: SSH from cp1 to the workers (a user with sudo) for temporary routes.

Proxmox, VMware, a cloud or Multipass – it does not matter. On a Mac with Apple Silicon, Multipass VMs are arm64. Linkerd and all images in the book are available for amd64 and arm64, and all downloads detect the architecture themselves (ARCH=$(dpkg --print-architecture)). The output in the book comes from amd64; file names and checksums then show arm64 on your machine.

Edge and stable

The Linkerd open source project only publishes edge releases, roughly every one to two weeks, named after the pattern edge-YY.MM.N. The Helm charts carry the same number as YYYY.MM.N: edge-26.11.2 corresponds to chart 2026.11.2. Stable versions with numbers like 2.20.x and long-term support are built by vendors themselves, for example Buoyant with “Buoyant Enterprise for Linkerd” (account required).

Edge does not mean “unstable”, but every version can change behavior. That is why the book pins edge-26.11.2 everywhere. Only Day 14 deliberately installs the previous version edge-26.11.1 on west, so that you practice a real upgrade including rollback on Day 15. The steps – read the release notes, CRDs first, then the Control Plane, then the Data Plane – are the same for a stable version.

Versions

Component Version from Day
Ubuntu (all VMs) 24.04 LTS 0
Kubernetes (ost, kubeadm) v1.37.2 0
Cilium 1.20.5 0
Helm v4.3.2 0
notes-api 0.7.1 0
go-httpbin (helper) 2.26.0 0
nginx-unprivileged (web) 1.31.7-alpine 0
postgres 18.7-alpine 0
curlimages/curl (last) 8.23.0 0
Linkerd (CLI, Control Plane, viz, multicluster) edge-26.11.2 1
Gateway API CRDs (from Cilium or k3s) v1.6.3 1
nicolaka/netshoot v0.15 3
kube-prometheus-stack (Helm chart) 93.4.1 4
step (Smallstep CLI) 0.31.0 5
cert-manager v1.21.3 5
cmctl v2.6.2 5
trust-manager v0.26.0 5
notes-api (old version for the canary) 0.5.2 9
k3s (west) v1.37.2+k3s1 12
Linkerd Helm charts (upgrade from → to) 2026.11.1 → 2026.11.2 14
busybox (banner service) 1.37 16
fortio/fortio (load generator) 1.69.5 17

The companion repo

The source code of notes-api and the material for the book live at https://github.com/thomaszachmann/linkerd-zum-anfassen. You do not need to build anything: the image ghcr.io/thomaszachmann/notes-api:0.7.1 is ready-made for amd64 and arm64. You still type the manifests and commands in the book yourself – that is the training.

What you can do afterwards

  • Install Linkerd with the CLI and Helm, check it, run it highly available, and upgrade and roll it back without a single failed request.
  • Bring workloads into the mesh and recognize when a container, a port or a protocol bypasses the proxy.
  • Prove mTLS on the wire, rotate the trust anchor and issuer, and hand over renewal to cert-manager.
  • Read success rate, throughput and latency of every service with viz, Prometheus and Grafana, and follow individual requests live.
  • Enforce default deny with Servers and AuthorizationPolicies without breaking probes and monitoring.
  • Control retries, timeouts, circuit breaking, rate limits, canary releases, ingress and egress with Gateway API resources, and measure their cost.
  • Connect two clusters with a shared trust anchor, mirror or federate services, and survive a cluster outage.
  • Find typical mesh failures in minutes with linkerd check, linkerd diagnostics and the proxy log.

Note: This book is an independent practice book. Linkerd is a CNCF project; Linkerd is a trademark of the Linux Foundation. The book is not affiliated with the Linux Foundation, the CNCF or vendors of Linkerd distributions and is neither reviewed nor endorsed by them.

All output comes from real runs, shortened where it makes sense. Values such as the API key lab-key-123, the database password lab-db-123 or the Grafana password lab-grafana-123 are lab values only – in production they belong in a password vault or a Secret, never in a book. The same goes for the private key of the trust anchor that you generate on Day 5.

Enjoy your drills.