Skip to content
Hands-on IT IT Drills

Day 15: Upgrade and Rollback

Goal: west runs edge-26.11.2 via Helm, the upgrade went through without a single failed request, and you have rolled the Control Plane back and forward again once. · Duration: approx. 75 min · Prerequisite: Day 14

Briefing

  • The order is fixed: CLI → CRDs → Control Plane → Data Plane → extensions. Each step tolerates the old state of the next one.
  • Helm upgrades the Control Plane. The proxies only switch when their Pods restart – that is your job, via rollout restart.
  • Beforehand, you read the release notes and compare the rendered manifests. What you do not find there should not happen in the cluster either.
  • helm rollback resets the Control Plane to an earlier revision. CRDs and proxies stay as they are.
 CLI 26.11.2 ─> linkerd-crds 2026.11.2 ─> linkerd-control-plane 2026.11.2
   ─> rollout restart (proxies 26.11.2) ─> linkerd-multicluster 2026.11.2
 meanwhile: last ─> helper in tag15, every request logged

Drill 1: Starting Point and Measurement

The app from Day 14 serves as the measurement object, at a higher rate (pause 0.2 instead of 0.5 s):

mkdir -p ~/linkerd/tag15 && cd ~/linkerd/tag15
sed -e 's/tag14/tag15/g' -e 's/sleep 0.5/sleep 0.2/' ../tag14/app.yaml > app.yaml
kubectl --context west apply -f app.yaml
kubectl --context west -n tag15 rollout status deploy --timeout=180s

Expected (abridged):

namespace/tag15 created
...
deployment "helper" successfully rolled out
deployment "last" successfully rolled out

Step 1 of the upgrade is the CLI. It has been in /usr/local/bin as edge-26.11.2 since Day 1. From now on, you use linkerd again for west instead of ./linkerd-edge-26.11.1. What does the new CLI see?

linkerd --context west version
linkerd --context west check 2>&1 | grep -A1 '^‼ control plane and cli'
linkerd --context west check --proxy -n tag15 2>&1 | grep -A3 '^‼ data plane is'

Expected:

Client version: edge-26.11.2
Server version: edge-26.11.1
‼ control plane and cli versions match
    control plane running edge-26.11.1 but cli running edge-26.11.2
‼ data plane is up-to-date
    some proxies are not running the current version:
    * helper-79d57fbbbb-bvxls (edge-26.11.1)
    * helper-79d57fbbbb-zg98l (edge-26.11.1)

A newer CLI against an older Control Plane is allowed and intended – the reverse is not. check only warns (‼) but ends with √.

Save your progress? Sign up for free

Drill 2: Read the Release Notes

curl -fsSL https://api.github.com/repos/linkerd/linkerd2/releases/tags/edge-26.11.2 \
  | jq -r '.published_at, .body' | grep -vE '^\* (ci|build|chore)' | head -5

Expected (abridged):

2026-11-13T23:50:48Z
## What's Changed
* fix(inject): inherit beta native sidecar annotation from namespace by @spicyshar…
* proxy: v2.373.0 by @l5d-bot in https://github.com/linkerd/linkerd2/pull/15768
* proxy: v2.374.0 by @l5d-bot in https://github.com/linkerd/linkerd2/pull/15781

What you look for:

  • Behavior changes that affect you. Here: the injector now also inherits config.beta.linkerd.io/proxy-enable-native-sidecar from the namespace. If you set the annotation on the namespace (Day 2 showed it on the Pod), you get different Pods after the restart.
  • Proxy versions: two proxy jumps are packed into one edge version.
  • CRDs, values, removed features, notes such as “breaking” or “action required”. If you skip versions, you read the notes of all versions in between. The monthly “Edge Release Roundup” on the Linkerd blog summarizes them.

Then you check what changes in the charts:

diff <(helm show values linkerd-edge/linkerd-control-plane --version 2026.11.1) \
  <(helm show values linkerd-edge/linkerd-control-plane --version 2026.11.2)
diff <(helm template x linkerd-edge/linkerd-crds --version 2026.11.1 \
  | grep -v helm.sh/chart) \
  <(helm template x linkerd-edge/linkerd-crds --version 2026.11.2 \
  | grep -v helm.sh/chart) && echo crds-gleich

Expected:

23c23
< linkerdVersion: edge-26.11.1
---
> linkerdVersion: edge-26.11.2
crds-gleich

Only the version changes; the CRDs are identical except for the chart label. A small step – this is exactly what upgrades should look like.

Save your progress? Sign up for free

Drill 3: Preview the Manifest

helm --kube-context west get manifest linkerd-control-plane -n linkerd > alt.yaml
helm template linkerd-control-plane linkerd-edge/linkerd-control-plane \
  --version 2026.11.2 -n linkerd \
  -f ../tag14/values-ha.yaml -f ../tag14/values-west.yaml > neu.yaml
diff alt.yaml neu.yaml | grep -E '^[<>] .*image:' | sort | uniq -c

Expected:

      1 <             image: cr.l5d.io/linkerd/controller:edge-26.11.1
      5 <         image: cr.l5d.io/linkerd/controller:edge-26.11.1
      6 <         image: cr.l5d.io/linkerd/proxy:edge-26.11.1
      1 >             image: cr.l5d.io/linkerd/controller:edge-26.11.2
      5 >         image: cr.l5d.io/linkerd/controller:edge-26.11.2
      6 >         image: cr.l5d.io/linkerd/proxy:edge-26.11.2

New images for the controllers (including the heartbeat CronJob) and for linkerd-init and linkerd-proxy in the three Control Plane Deployments – both use the proxy image. The rest of the diff consists of version labels and the webhook certificates, which every render regenerates.

Warning: No --reuse-values. With it, Helm keeps your old values but ignores new chart defaults. Pass the same values files with every upgrade.

Save your progress? Sign up for free

Drill 4: CRDs, Then Control Plane

Record the start time, then both releases one after the other:

date -u +%FT%TZ > start.txt
helm --kube-context west upgrade linkerd-crds linkerd-edge/linkerd-crds \
  --version 2026.11.2 -n linkerd
helm --kube-context west upgrade linkerd-control-plane \
  linkerd-edge/linkerd-control-plane --version 2026.11.2 -n linkerd \
  -f ../tag14/values-ha.yaml -f ../tag14/values-west.yaml
kubectl --context west -n linkerd rollout status deploy --timeout=300s
kubectl --context west -n linkerd get pods

Expected (abridged):

STATUS: deployed
REVISION: 2
...
STATUS: deployed
REVISION: 3
Waiting for deployment "linkerd-destination" rollout to finish: 1 old replicas are …
deployment "linkerd-destination" successfully rolled out
deployment "linkerd-identity" successfully rolled out
deployment "linkerd-proxy-injector" successfully rolled out
NAME                                      READY   STATUS        RESTARTS   AGE
linkerd-destination-6fccb5879-k944c       4/4     Running       0          11s
linkerd-destination-6fccb5879-lvqmp       4/4     Running       0          12s
linkerd-destination-755db578f5-62kmw      4/4     Terminating   0          4m14s
linkerd-identity-74946cc965-6sh5z         2/2     Running       0          11s
...

About 20 seconds for all three Deployments. maxUnavailable: 1 from the HA values always keeps one replica up. And the traffic?

sleep 20
kubectl --context west -n tag15 logs deploy/last -c last \
  --since-time=$(cat start.txt) \
  | awk '{print $2, $3}' | sort | uniq -c

Expected:

    188 /get 200

Not a single error. The proxies keep their connections and endpoints, even while destination is switching.

Save your progress? Sign up for free

Drill 5: Catch Up the Data Plane

linkerd --context west version --short
linkerd --context west check --proxy -n tag15 2>&1 | grep -E '^‼|^\s+\*'

Expected:

edge-26.11.2
edge-26.11.2
‼ data plane is up-to-date
    * helper-79d57fbbbb-bvxls (edge-26.11.1)
    * helper-79d57fbbbb-zg98l (edge-26.11.1)
    * last-55f7c55d84-lg69g (edge-26.11.1)
‼ data plane and cli versions match

The Control Plane is new, the proxies are old. The injector only puts the new version into new Pods. The server first, so that last keeps measuring meanwhile:

date -u +%FT%TZ > start2.txt
kubectl --context west -n tag15 rollout restart deploy/helper
kubectl --context west -n tag15 rollout status deploy/helper
sleep 20
kubectl --context west -n tag15 logs deploy/last -c last \
  --since-time=$(cat start2.txt) \
  | awk '{print $2, $3}' | sort | uniq -c

Expected (abridged):

deployment.apps/helper restarted
...
deployment "helper" successfully rolled out
    128 /get 200

Then the client itself. Its log starts fresh afterwards:

kubectl --context west -n tag15 rollout restart deploy/last
kubectl --context west -n tag15 rollout status deploy/last
sleep 30
kubectl --context west -n tag15 get pods -o \
  custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd --context west check --proxy -n tag15 2>&1 | grep -E 'data plane is|Status'

Expected:

deployment.apps/last restarted
deployment "last" successfully rolled out
NAME                      PROXY
helper-5d777bbb87-7bwcn   edge-26.11.2
helper-5d777bbb87-nn2dn   edge-26.11.2
last-79f6974c-gjpnz       edge-26.11.2
√ data plane is up-to-date
Status check results are √

Tip: Right after the rollout, check --proxy still reports the old last Pod. The curl loop does not react to SIGTERM and is only terminated after 30 seconds (terminationGracePeriodSeconds).

Save your progress? Sign up for free

Drill 6: Roll Back the Control Plane

Suppose the new version causes trouble. Helm knows every revision:

date -u +%FT%TZ > start3.txt
helm --kube-context west history linkerd-control-plane -n linkerd
helm --kube-context west rollback linkerd-control-plane 2 -n linkerd
kubectl --context west -n linkerd rollout status deploy --timeout=300s | tail -1
helm --kube-context west history linkerd-control-plane -n linkerd | tail -2
linkerd --context west version --short

Expected (abridged):

REVISION    UPDATED                     STATUS      CHART                 … DESCRIPTION
1           Thu Nov 19 20:04:09 2026    superseded  linkerd-control-plane…  Install complete
2           Thu Nov 19 20:08:41 2026    superseded  linkerd-control-plane…  Upgrade complete
3           Thu Nov 19 20:19:16 2026    deployed    linkerd-control-plane…  Upgrade complete
Rollback was a success! Happy Helming!
deployment "linkerd-proxy-injector" successfully rolled out
3           Thu Nov 19 20:19:16 2026    superseded  … edge-26.11.2  Upgrade complete
4           Thu Nov 19 20:21:32 2026    deployed    … edge-26.11.1  Rollback to 2
edge-26.11.2
edge-26.11.1

Revision 2 is the state before the upgrade (Day 14 after the anti-affinity fix). The rollback does not overwrite history; it creates revision 4 as a copy of 2.

sleep 10
kubectl --context west -n tag15 logs deploy/last -c last \
  --since-time=$(cat start3.txt) \
  | awk '{print $2, $3}' | sort | uniq -c
linkerd --context west check --proxy -n tag15 2>&1 | grep -E '^‼'

Expected:

    164 /get 200
‼ control plane proxies are up-to-date
‼ control plane proxies and cli versions match

Again no errors. What is interesting is what is not reported: the app proxies now run edge-26.11.2 against a Control Plane on edge-26.11.1. check only compares with the CLI and the latest version, not proxy with Control Plane. For one step this works fine; for larger gaps, you restart the Data Plane after the rollback as well. The CRDs stay at 2026.11.2 – a separate release, and you do not roll back CRDs lightly: older schemas can drop fields of newer objects.

Save your progress? Sign up for free

Drill 7: Forward Again

Going forward uses the same command as in Drill 4, not rollback 3. That way, the values file remains the single source of truth:

date -u +%FT%TZ > start4.txt
helm --kube-context west upgrade linkerd-control-plane \
  linkerd-edge/linkerd-control-plane --version 2026.11.2 -n linkerd \
  -f ../tag14/values-ha.yaml -f ../tag14/values-west.yaml
kubectl --context west -n linkerd rollout status deploy --timeout=300s | tail -1
helm --kube-context west list -n linkerd
sleep 15
kubectl --context west -n tag15 logs deploy/last -c last \
  --since-time=$(cat start4.txt) \
  | awk '{print $2, $3}' | sort | uniq -c
linkerd --context west check --proxy -n tag15 2>&1 | tail -1

Expected (abridged):

REVISION: 5
deployment "linkerd-proxy-injector" successfully rolled out
NAME                    NAMESPACE   REVISION    … CHART                             APP VERSION
linkerd-control-plane   linkerd     5           … linkerd-control-plane-2026.11.2   edge-26.11.2
linkerd-crds            linkerd     2           … linkerd-crds-2026.11.2
    150 /get 200
Status check results are √

Upgrade, Data Plane restart, rollback, and another upgrade: in the measured windows, about 630 requests, zero errors. That is the reward for two replicas, a PDB, and maxUnavailable: 1.

Save your progress? Sign up for free

Drill 8: The Multicluster Extension via Helm

Extensions come after the Control Plane, in the same version. The probe port gets a fixed place in the values. Day 12 showed why: the link on the other side and the probe Service must name the same port.

cat > mc-values.yaml <<'EOF'
gateway:
  serviceType: LoadBalancer   # k3s ServiceLB provides the VM's IP
  probe:
    port: 4191                # the other side probes this port (Day 12)
EOF
helm --kube-context west install linkerd-multicluster \
  linkerd-edge/linkerd-multicluster --version 2026.11.2 \
  -n linkerd-multicluster --create-namespace -f mc-values.yaml
kubectl --context west -n linkerd-multicluster rollout status deploy --timeout=180s
kubectl --context west get ns linkerd-multicluster --show-labels
kubectl --context west -n linkerd-multicluster get svc linkerd-gateway

Expected (abridged):

STATUS: deployed
REVISION: 1
deployment "linkerd-gateway" successfully rolled out
deployment "linkerd-local-service-mirror" successfully rolled out
NAME                   STATUS   AGE   LABELS
linkerd-multicluster   Active   11s   …,linkerd.io/extension=multicluster,…
NAME              TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)
linkerd-gateway   LoadBalancer   10.43.97.226   172.16.0.182   4143:31923/TCP,4191:32…

Here --create-namespace is enough: a Job of the chart (hook post-install) sets the label linkerd.io/extension=multicluster, by which linkerd check recognizes the extension. The Link CRD is in the same chart. For ost to read west as on Day 12, nothing else is needed on west. If west is to read ost later, you add an entry under controllers with gateway.probe.port: 30980 (NodePort on ost) – before the link.

linkerd --context west multicluster check 2>&1 | tail -5
linkerd --context west check 2>&1 | tail -1

Expected:

√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are √
Status check results are √

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the proxy version for every Pod in tag15. Target time: 30 s
  2. Find the chart version for edge-26.10.2. Target time: 20 s
  3. Show all Pods on west whose proxy is not edge-26.11.2. Target time: 60 s
  4. Show which revision of the Control Plane is currently active. Target time: 15 s

Solution

kubectl --context west -n tag15 get pods -o \
  custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
helm search repo linkerd-edge/linkerd-control-plane --versions | grep edge-26.10.2
kubectl --context west get pods -A -l linkerd.io/control-plane-ns=linkerd -o \
  custom-columns=NS:.metadata.namespace,NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version | grep -v edge-26.11.2
helm --kube-context west history linkerd-control-plane -n linkerd | grep deployed

Expected (abridged, task 3 during the Challenge):

NAME                      PROXY
helper-5d777bbb87-7bwcn   edge-26.11.2
...
linkerd-edge/linkerd-control-plane  2026.10.2       edge-26.10.2    Linkerd gives you…
NS        NAME                                      PROXY
tag15     last-748d866d8c-8h9px                     edge-26.11.1
5           Thu Nov 19 20:22:22 2026    deployed    linkerd-control-plane-2026.11.2 …

Save your progress? Sign up for free

Checkpoint

helm --kube-context west list -A -o json | jq -r '.[] | .name + " " + .chart'

Expected:

gateway-api-crd gateway-api-crd-1.6.301
linkerd-control-plane linkerd-control-plane-2026.11.2
linkerd-crds linkerd-crds-2026.11.2
linkerd-multicluster linkerd-multicluster-2026.11.2
linkerd --context west check 2>&1 | tail -1
linkerd --context west multicluster check 2>&1 | tail -1

Expected: Status check results are √ twice.

kubectl --context west get ns

Expected: after cleanup, no tag14 and no tag15, but linkerd and linkerd-multicluster.

Challenge

You want to see the new proxy version on just one workload first – or keep a workload on the old one temporarily. Pin last to edge-26.11.1, show the mixed Data Plane, and remove the pin again.

Hint: The proxy version can be chosen per Pod template via an annotation.

Solution

kubectl --context west -n tag15 patch deploy last -p \
  '{"spec":{"template":{"metadata":{"annotations":{
  "config.linkerd.io/proxy-version":"edge-26.11.1"}}}}}'
kubectl --context west -n tag15 rollout status deploy/last
sleep 40
kubectl --context west -n tag15 get pods -o \
  custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
kubectl --context west -n tag15 logs deploy/last -c last --since=30s \
  | awk '{print $2, $3}' | sort | uniq -c

Expected:

deployment.apps/last patched
deployment "last" successfully rolled out
NAME                      PROXY
helper-5d777bbb87-7bwcn   edge-26.11.2
helper-5d777bbb87-nn2dn   edge-26.11.2
last-748d866d8c-8h9px     edge-26.11.1
    125 /get 200

Old client, new server, no errors. Remove the pin (~1 stands for / in the JSON path):

kubectl --context west -n tag15 patch deploy last --type=json -p '[{"op":"remove",
  "path":"/spec/template/metadata/annotations/config.linkerd.io~1proxy-version"}]'
kubectl --context west -n tag15 rollout status deploy/last

Expected:

deployment.apps/last patched
deployment "last" successfully rolled out

The same principle underpins the production strategy: after upgrading the Control Plane, you first restart a canary namespace and watch it before the rest follows. The annotation on the namespace keeps teams that are not ready yet on the old version for a while.

Save your progress? Sign up for free

Cleanup

kubectl --context west delete namespace tag15
rm -f ../tag14/linkerd-edge-26.11.1 alt.yaml neu.yaml start*.txt
kubectl --context west get ns

Expected:

namespace "tag15" deleted
NAME                   STATUS   AGE
default                Active   5h4m
kube-node-lease        Active   5h4m
kube-public            Active   5h4m
kube-system            Active   5h4m
linkerd                Active   22m
linkerd-multicluster   Active   71s

west now runs permanently via Helm with edge-26.11.2. For a new link ost → west, you install the controller on ost as on Day 12, Drill 5, but with ../tag12/mc-ost-gw.yaml (otherwise the NodePort gateway disappears), and create the link as in Drill 6 – link-gen still runs via CLI, unchanged. From now on, you change the extension on west only via helm upgrade with mc-values.yaml.

Interview Note

  • In what order do you upgrade Linkerd? CLI, CRDs, Control Plane, then the Data Plane via restart, and finally the extensions. Release notes and diff beforehand.
  • What does helm rollback do – and what does it not do? It applies an earlier revision of the Control Plane as a new revision. CRDs (a separate release) and running proxies stay unchanged.
  • How do you notice that proxies are outdated? linkerd check --proxy and the annotation linkerd.io/proxy-version on every Pod.

Enterprise Note

  • Keep up with edge regularly, in small steps. Vendors’ stable lines only minor by minor (2.19 → 2.20); when skipping edge versions, read all release notes.
  • First a staging environment, then in production a canary namespace with a load generator and an SLO dashboard, and only then the rest of the Data Plane.
  • Do not leave proxies old across several upgrades. Testing mainly covers a new Control Plane with proxies of the immediately preceding version.
  • Stagger Data Plane restarts (namespace by namespace) and keep the apps’ PDBs in mind: single replicas such as databases go down briefly with every upgrade.
  • Upgrades via GitOps: bumping the chart version in Git is the change, reverting it is the rollback. Otherwise, a manual helm rollback creates drift from the Git state.