Skip to content
Hands-on IT IT Drills

Day 2: Bringing Pods into the Mesh

Goal: The whole app runs with the Linkerd proxy, every service has its own identity, and you know when a container bypasses the mesh. · Duration: approx. 60 min · Prerequisite: Day 1

Briefing

  • The proxy gets into the Pod via a webhook, at creation time. Running Pods do not change – you have to restart them.
  • You control this with the annotation linkerd.io/inject on the namespace or on the Pod template. The Pod annotation wins.
  • A proxy’s identity is its ServiceAccount. If all services share default, Linkerd cannot tell them apart later.
  • Protocols where detection cannot decide, such as Postgres or MySQL, need opaque ports.
 Pod notes-api
 ┌──────────────────────────────────────────────────────┐
 │ init: linkerd-init   iptables: everything via proxy  │
 │ init: linkerd-proxy  native sidecar, keeps running   │
 │ init: migrate        already talks through proxy     │
 │       notes-api      the app                         │
 └──────────────────────────────────────────────────────┘

Drill 1: Mark the namespace

mkdir -p ~/linkerd/tag02 && cd ~/linkerd/tag02
kubectl annotate namespace notes linkerd.io/inject=enabled
kubectl -n notes get pods

Expected (abridged):

namespace/notes annotated
NAME                         READY   STATUS    RESTARTS   AGE
helper-564786ddcd-2dxzp      1/1     Running   0          3h50m
...
web-84fb8799ff-nl4jl         1/1     Running   0          3h47m

Nothing happens: the webhook only applies to new Pods. So restart:

kubectl -n notes rollout restart deploy/notes-api deploy/helper deploy/web deploy/last
kubectl -n notes rollout restart statefulset/postgres
kubectl -n notes rollout status deploy/notes-api --timeout=180s
kubectl -n notes rollout status statefulset/postgres --timeout=180s
kubectl -n notes get pods

Expected (abridged):

NAME                         READY   STATUS    RESTARTS   AGE
helper-54d9fbb894-4dhxf      2/2     Running   0          13s
helper-54d9fbb894-7mwbz      2/2     Running   0          25s
last-5d56df4dc9-j7fn4        2/2     Running   0          25s
notes-api-745cf7dd7c-d778n   2/2     Running   0          25s
notes-api-745cf7dd7c-m9fl8   2/2     Running   0          10s
postgres-0                   2/2     Running   0          24s
web-85577d96fd-5q5zk         2/2     Running   0          25s
web-85577d96fd-ptnnm         2/2     Running   0          13s

2/2: app plus proxy. The proxy counts as a container, even though technically it is an init container that keeps running.

Save your progress? Sign up for free

Drill 2: What the webhook changed

kubectl -n notes get pod -l app=notes-api \
  -o jsonpath='{.items[0].spec.initContainers[*].name}{"\n"}'
kubectl -n notes get pod -l app=notes-api \
  -o jsonpath='{.items[0].spec.initContainers[*].restartPolicy}{"\n"}'
kubectl -n notes get pod -l app=notes-api \
  -o jsonpath='{.items[0].metadata.annotations}' \
  | tr ',' '\n' | grep linkerd
kubectl -n notes logs deploy/notes-api -c migrate | tail -1

Expected (abridged):

linkerd-init linkerd-proxy migrate
Always
"linkerd.io/created-by":"linkerd/proxy-injector edge-26.11.2"
"linkerd.io/inject":"enabled"
"linkerd.io/proxy-version":"edge-26.11.2"
"linkerd.io/trust-root-sha256":"1f3f4f01141d3508be8970d5ecc349ff3a2381b5805f4e..."
time=2026-11-19T15:06:23.635Z level=INFO msg="migration complete" table=notes

The order matters: linkerd-init sets the iptables rules, then the proxy starts as a native sidecar, and only then your init container migrate. Its connection to Postgres therefore already runs through the mesh.

Save your progress? Sign up for free

Drill 3: Without a native sidecar

Older Linkerd versions and clusters without native sidecars behave differently. You can turn on the old behavior per Pod:

kubectl create namespace tag02
kubectl annotate namespace tag02 linkerd.io/inject=enabled
kubectl -n tag02 create secret generic notes-db --from-literal=url=\
'postgres://notes:[email protected]:5432/notes?sslmode=disable'
cat > migrate-alt.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: migrate-alt, namespace: tag02}
spec:
  replicas: 1
  selector: {matchLabels: {app: migrate-alt}}
  template:
    metadata:
      labels: {app: migrate-alt}
      annotations:
        config.beta.linkerd.io/proxy-enable-native-sidecar: "false"
    spec:
      initContainers:
      - name: migrate
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        args: [migrate]
        env:
        - name: NOTES_DB_URL
          valueFrom: {secretKeyRef: {name: notes-db, key: url}}
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.10
EOF
kubectl apply -f migrate-alt.yaml
sleep 30
P=$(kubectl -n tag02 get pod -l app=migrate-alt -o name)
kubectl -n tag02 get $P -o jsonpath='{.spec.initContainers[*].name}{"\n"}'
kubectl -n tag02 logs $P -c migrate --tail=1

Expected:

migrate linkerd-init
time=2026-11-19T15:06:55.244Z level=INFO msg="migration complete" table=notes

migrate now runs before linkerd-init; the proxy is a regular container and only starts afterwards. So migrate runs entirely without a proxy: unencrypted and without an identity. Today that works. As soon as Postgres only accepts connections from the mesh on Day 6, exactly this container fails. Stick with native sidecars.

Save your progress? Sign up for free

Drill 4: Opaque ports

linkerd check --proxy -n notes

Expected (abridged):

linkerd-data-plane
------------------
√ data plane namespace exists
√ data plane proxies are ready
...
‼ opaque ports are properly annotated
    * service postgres targets the opaque port postgres through 5432; add 5432 to its
      config.linkerd.io/opaque-ports annotation

With Postgres, the client sends only a short startup message and then waits for the server. The proxy cannot recognize a protocol from that and waits until the timeout of 10 seconds. On an opaque port, it treats the connection as a TCP stream right away. Linkerd already knows this for port 5432:

kubectl -n linkerd get configmap linkerd-config -o yaml | grep -o 'opaquePorts:.*'

Expected:

opaquePorts: 25,587,3306,4444,5432,6379,9300,11211

The Service, however, refers to the named port postgres. linkerd check still demands the annotation on the Service – so annotate it explicitly:

kubectl -n notes annotate service postgres config.linkerd.io/opaque-ports=5432
linkerd check --proxy -n notes 2>&1 | grep -A2 opaque

Expected:

√ opaque ports are properly annotated

Status check results are √

Save your progress? Sign up for free

Drill 5: One identity per service

kubectl -n notes logs deploy/web -c linkerd-proxy | grep -E 'release|identity'

Expected (abridged):

INFO ThreadId(01) linkerd2_proxy: release 2.374.0 (16d3717) by linkerd on 2026-11-11...
INFO ThreadId(01) linkerd2_proxy: SNI is default.notes.serviceaccount.identity.link...
INFO ThreadId(01) linkerd2_proxy: Local identity is default.notes.serviceaccount.ide...

All services are called default.notes – Linkerd cannot tell web from helper. So every service gets its own ServiceAccount:

for a in web notes-api helper last postgres; do
  kubectl -n notes create serviceaccount $a
done
for d in web notes-api helper last; do
  kubectl -n notes set serviceaccount deploy/$d $d
done
kubectl -n notes set serviceaccount statefulset/postgres postgres
kubectl -n notes rollout status deploy/web --timeout=180s
kubectl -n notes rollout status statefulset/postgres --timeout=180s
kubectl -n notes logs deploy/web -c linkerd-proxy | grep 'Local identity'

Expected (abridged):

INFO ThreadId(01) linkerd2_proxy: Local identity is web.notes.serviceaccount.identity...

Warning: From now on, do not apply the manifests from ~/linkerd/tag00 anymore – they do not know about the ServiceAccounts.

Save your progress? Sign up for free

Drill 6: Inject by hand and exclude

The CLI can prepare a manifest. Without an option it only sets the annotation; with --manual it writes the proxy directly into it:

kubectl -n tag02 create deployment plain --image=registry.k8s.io/pause:3.10 \
  --dry-run=client -o yaml > plain.yaml
linkerd inject plain.yaml | grep -A2 'annotations'
linkerd inject --manual plain.yaml | grep -E 'name: linkerd-(init|proxy)$'

Expected:

deployment "plain" injected

      annotations:
        linkerd.io/inject: enabled
      labels:
deployment "plain" injected

        name: linkerd-init
        name: linkerd-proxy

The annotation route is better: on upgrades, the Pod gets the new proxy at its next restart without you regenerating manifests.

You can exclude per Pod:

kubectl -n tag02 create deployment ohne --image=registry.k8s.io/pause:3.10
kubectl -n tag02 patch deploy ohne -p \
  '{"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"disabled"}}}}}'
kubectl -n tag02 rollout status deploy/ohne
kubectl -n tag02 get pods -l app=ohne

Expected:

NAME                  READY   STATUS    RESTARTS   AGE
ohne-58b54f56-fs5bd   1/1     Running   0          15s

Save your progress? Sign up for free

Drill 7: Has the mesh changed anything?

sleep 60
kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (numbers vary):

     92 /api/notes 200
     29 /flaky 200
     17 /flaky 500

Same error rate as on Day 0. Linkerd changes nothing about the behavior as long as you configure nothing. Note -c last: the Pod now has two containers.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show which Pods in notes have a proxy. Target time: 30 s
  2. Show the proxy version in notes. Target time: 20 s
  3. Show the first log lines of the helper proxy. Target time: 20 s
  4. Show which namespaces are annotated and how. Target time: 40 s

Solution

kubectl -n notes get pods -o custom-columns=\
NAME:.metadata.name,PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd version --proxy -n notes
kubectl -n notes logs deploy/helper -c linkerd-proxy | head -5
kubectl get ns -o \
  custom-columns=NAME:.metadata.name,INJECT:.metadata.annotations.linkerd\\.io/inject

Save your progress? Sign up for free

Checkpoint

kubectl -n notes get pods

Expected: all Pods 2/2 Running.

linkerd check --proxy -n notes 2>&1 | tail -1

Expected:

Status check results are √
kubectl -n notes get deploy notes-api \
  -o jsonpath='{.spec.template.spec.serviceAccountName}'

Expected: notes-api

Challenge

Change tag02 so that by default no proxy is injected there, but a new Deployment mit still gets one.

Hint: The Pod annotation wins over the namespace annotation.

Solution

kubectl annotate namespace tag02 linkerd.io/inject=disabled --overwrite
kubectl -n tag02 create deployment mit --image=registry.k8s.io/pause:3.10
kubectl -n tag02 patch deploy mit -p \
  '{"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"enabled"}}}}}'
kubectl -n tag02 rollout status deploy/mit
kubectl -n tag02 get pods -l app=mit

Expected:

NAME                   READY   STATUS    RESTARTS   AGE
mit-56456f7865-t7gxm   2/2     Running   0          6s

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag02

Interview Note

  • How does the proxy get into the Pod? A mutating webhook adds linkerd-init and linkerd-proxy at creation time. Existing Pods need a restart.
  • Where does a workload’s identity come from? From its ServiceAccount – that is why every service needs its own.
  • What are opaque ports? Ports on which the proxy does not detect a protocol but passes TCP through – for example with databases such as Postgres or MySQL, where detection would otherwise hang for up to 10 s.

Enterprise Note

  • Turn on injection per namespace, document exceptions per workload.
  • Jobs and CronJobs: the native sidecar ends with the Job. With old sidecars, Jobs got stuck – a common reason for frustration with service meshes.
  • Admission policies (Kyverno, Gatekeeper) can enforce that Pods never use the ServiceAccount default.
  • Schedule restarts for injection in maintenance windows and protect them with a PodDisruptionBudget.