Preface
A service mesh promises a lot: mTLS between all services without touching the app, success rates and latencies for every connection, retries, timeouts, canary releases, identity-based policies, and services that are reachable across cluster boundaries. Whether that holds true, you only find out in operation. Then an init container suddenly bypasses the proxy, a retry triples the load on a service that is already down, the issuer expires at night, or a single wrong ServiceAccount locks out an entire chain with 403. This book prepares you for exactly these moments – not with slides, but in the terminal.
You run Linkerd in a real cluster, on a small app made of four
services that makes mistakes on purpose. Every feature starts with a
measurement without it and ends with a measurement with it: success rate
in the load generator, linkerd viz stat, Prometheus, a
capture with tcpdump. What you cannot prove, you have not
configured.
The book is aimed at platform and DevOps engineers with Kubernetes
basics: you should know Deployments, Services and ServiceAccounts, read
YAML, use kubectl and be able to install a Helm chart. You
should roughly know how TLS certificates work in principle; everything
Linkerd builds on top of that, you learn here. You do not need to have
run a service mesh before.
How the book works
- One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
- Before, configure, after. Every feature ends with proof: a line in the load generator, a metric from the proxy, a packet on the wire.
- No helper script. Every step is shown as a single command, including the Checkpoint at the end of each day. You write manifests to files with heredocs.
- Speed Drills at the end of each day: small tasks with a target time.
- Checkpoint, Challenge, Interview Note, Enterprise Note close out every day. The Checkpoint uses single commands to check whether your cluster is in the expected state – the following days build on it. The Challenge is a task without instructions, with a hint and a solution. The Interview Note collects the questions that come up in a job interview or on call, along with the core answer. The Enterprise Note tells you what is different in production than in the lab.
- One app through all days:
noteswithweb(nginx),notes-api(Go),helper(go-httpbin, fails 30 percent of calls) andpostgres. The load generatorlastwrites every status code to its log – that is your measuring instrument. Days that break something work in a copy in their own namespace.
To finish:
- Competition (Day 18): a broken mesh that you repair under time pressure.
- Appendix A is the cheat sheet, Appendix B the final exam with tasks across the whole book.
The days at a glance
| Day | Topic | Duration |
|---|---|---|
| 0 | Training ground: check the cluster, app notes, baseline
without mesh |
approx. 45 min |
| 1 | Install Linkerd: CLI, CRDs, Control Plane,
linkerd check |
approx. 45 min |
| 2 | Bring Pods into the mesh: injection, native sidecar, opaque ports | approx. 60 min |
| 3 | mTLS and identity: proxy certificates, proof with
tcpdump |
approx. 60 min |
| 4 | Observe: linkerd-viz, Prometheus, Grafana, tap |
approx. 75 min |
| 5 | Certificates: your own trust anchor, rotation, cert-manager | approx. 90 min |
| 6 | Authorization: Server, AuthorizationPolicy, default deny | approx. 75 min |
| 7 | Retries and timeouts with HTTPRoutes | approx. 60 min |
| 8 | Circuit breaking and rate limiting | approx. 60 min |
| 9 | Traffic shifting: canary, header routing, rollback | approx. 60 min |
| 10 | Ingress with Gateway API: Cilium gateway and proxy in the mesh | approx. 75 min |
| 11 | Egress: EgressNetwork, TLSRoute, HTTPRoute to the outside | approx. 60 min |
| 12 | Multicluster: second cluster, link, mirrored services | approx. 90 min |
| 13 | Multicluster: failover and federated services | approx. 90 min |
| 14 | Linkerd with Helm and highly available | approx. 75 min |
| 15 | Upgrade and rollback | approx. 75 min |
| 16 | Troubleshooting: eight typical mesh failures | approx. 90 min |
| 17 | What the mesh costs: latency, CPU, memory, proxy resources | approx. 90 min |
| 18 | Competition: the broken mesh | approx. 90 min |
Time: Days 0 to 16 add up to just under 20 hours (1185 minutes), Day 17 and the Competition add 180 minutes – just under 23 hours in total, plus the final exam. At one training day per evening and five evenings per week, you are through in just under four weeks.
The order is no accident: from Day 6 on, default deny applies in
notes, and every later day has to live with it. Day 12 only
requires Day 5 (the trust anchor is shared), Days 13 to 15 build on Day
12, Day 16 requires Day 7, Day 17 only Day 4.
The training ground
The book works with two clusters. ost is the main
cluster for all days, west joins on Day 12 for multicluster
and carries the Helm installation on Days 14 and 15. All commands run as
ubuntu on cp1, the working directory is
~/linkerd/tagNN/.
| Role | Purpose | IP in the book | In the lab |
|---|---|---|---|
| cp1 | Control Plane ost, workstation |
172.16.0.126 | 4 vCPU, 12 GB RAM |
| worker1 | Worker ost |
172.16.0.228 | 2 vCPU, 4 GB RAM |
| worker2 | Worker ost |
172.16.0.83 | 2 vCPU, 4 GB RAM |
| west | k3s single node (from Day 12) | 172.16.0.182 | 4 vCPU, 7 GB RAM |
ost: kubeadm 1.37.2 with three nodes, Cilium 1.20.5 as the CNI without kube-proxy, Gateway API with the Cilium GatewayClasscilium, a StorageClasslocal-pathand Helm. How to build such a cluster is shown in “CKA hands-on” and “Cilium hands-on”; if you still have the training ground, use it.west: another VM with Ubuntu 24.04, on which Day 12 installs k3s v1.37.2 as a single node.cp1needs SSH access towest(a user withsudo), and the Pod and Service networks must not overlap withost.- Minimum: The lab values are generous because
exercises from other books were also running on the VMs. For this book
alone, in our estimation, two workers with 2 vCPU and 4 GB each, a
Control Plane with 4 vCPU and 8 GB, and 2 vCPU and 4 GB for
westare enough. We have not tested anything smaller.
To be honest about what the days require:
- Day 0: Cilium’s socket load balancing must not rewrite Service addresses inside the Pod, otherwise routes and traffic shifting do not take effect. Drill 2 checks this. With Calico or Flannel, the drill is skipped.
- Day 4: you install kube-prometheus-stack yourself (release
kpsin the namespacemonitoring). If you already have one, substitute the release name and namespace. - Day 5: cert-manager v1.21.3. If it is not running, the installation
command is in the chapter. You also need
stepandcmctloncp1, plusjqfrom Day 4 on. - Day 10: a Gateway API controller. In the lab that is Cilium; without LoadBalancer IPs, you access it via NodePort.
- Day 11: Pods need access to the internet (
example.com,linkerd.io). - Day 13: SSH from
cp1to the workers (a user withsudo) for temporary routes.
Proxmox, VMware, a cloud or Multipass – it does not matter. On a Mac
with Apple Silicon, Multipass VMs are arm64. Linkerd and all images in
the book are available for amd64 and arm64, and all downloads detect the
architecture themselves
(ARCH=$(dpkg --print-architecture)). The output in the book
comes from amd64; file names and checksums then show arm64
on your machine.
Edge and stable
The Linkerd open source project only publishes edge
releases, roughly every one to two weeks, named after the
pattern edge-YY.MM.N. The Helm charts carry the same number
as YYYY.MM.N: edge-26.11.2 corresponds to
chart 2026.11.2. Stable versions with numbers like 2.20.x
and long-term support are built by vendors themselves, for example
Buoyant with “Buoyant Enterprise for Linkerd” (account required).
Edge does not mean “unstable”, but every version can change behavior.
That is why the book pins edge-26.11.2 everywhere. Only Day
14 deliberately installs the previous version edge-26.11.1
on west, so that you practice a real upgrade including
rollback on Day 15. The steps – read the release notes, CRDs first, then
the Control Plane, then the Data Plane – are the same for a stable
version.
Versions
| Component | Version | from Day |
|---|---|---|
| Ubuntu (all VMs) | 24.04 LTS | 0 |
Kubernetes (ost, kubeadm) |
v1.37.2 | 0 |
| Cilium | 1.20.5 | 0 |
| Helm | v4.3.2 | 0 |
| notes-api | 0.7.1 | 0 |
go-httpbin (helper) |
2.26.0 | 0 |
nginx-unprivileged (web) |
1.31.7-alpine | 0 |
postgres |
18.7-alpine | 0 |
curlimages/curl (last) |
8.23.0 | 0 |
| Linkerd (CLI, Control Plane, viz, multicluster) | edge-26.11.2 | 1 |
| Gateway API CRDs (from Cilium or k3s) | v1.6.3 | 1 |
nicolaka/netshoot |
v0.15 | 3 |
| kube-prometheus-stack (Helm chart) | 93.4.1 | 4 |
step (Smallstep CLI) |
0.31.0 | 5 |
| cert-manager | v1.21.3 | 5 |
cmctl |
v2.6.2 | 5 |
| trust-manager | v0.26.0 | 5 |
| notes-api (old version for the canary) | 0.5.2 | 9 |
k3s (west) |
v1.37.2+k3s1 | 12 |
| Linkerd Helm charts (upgrade from → to) | 2026.11.1 → 2026.11.2 | 14 |
busybox (banner service) |
1.37 | 16 |
fortio/fortio (load generator) |
1.69.5 | 17 |
The companion repo
The source code of notes-api and the material for the book live at
https://github.com/thomaszachmann/linkerd-zum-anfassen. You
do not need to build anything: the image
ghcr.io/thomaszachmann/notes-api:0.7.1 is ready-made for
amd64 and arm64. You still type the manifests and commands in the book
yourself – that is the training.
What you can do afterwards
- Install Linkerd with the CLI and Helm, check it, run it highly available, and upgrade and roll it back without a single failed request.
- Bring workloads into the mesh and recognize when a container, a port or a protocol bypasses the proxy.
- Prove mTLS on the wire, rotate the trust anchor and issuer, and hand over renewal to cert-manager.
- Read success rate, throughput and latency of every service with viz, Prometheus and Grafana, and follow individual requests live.
- Enforce default deny with Servers and AuthorizationPolicies without breaking probes and monitoring.
- Control retries, timeouts, circuit breaking, rate limits, canary releases, ingress and egress with Gateway API resources, and measure their cost.
- Connect two clusters with a shared trust anchor, mirror or federate services, and survive a cluster outage.
- Find typical mesh failures in minutes with
linkerd check,linkerd diagnosticsand the proxy log.
Note: This book is an independent practice book. Linkerd is a CNCF project; Linkerd is a trademark of the Linux Foundation. The book is not affiliated with the Linux Foundation, the CNCF or vendors of Linkerd distributions and is neither reviewed nor endorsed by them.
All output comes from real runs, shortened where it makes sense.
Values such as the API key lab-key-123, the database
password lab-db-123 or the Grafana password
lab-grafana-123 are lab values only – in production they
belong in a password vault or a Secret, never in a book. The same goes
for the private key of the trust anchor that you generate on Day 5.
Enjoy your drills.