Day 15: Upgrade and Rollback
Goal:
westrunsedge-26.11.2via Helm, the upgrade went through without a single failed request, and you have rolled the Control Plane back and forward again once. · Duration: approx. 75 min · Prerequisite: Day 14
Briefing
- The order is fixed: CLI → CRDs → Control Plane → Data Plane → extensions. Each step tolerates the old state of the next one.
- Helm upgrades the Control Plane. The proxies only
switch when their Pods restart – that is your job, via
rollout restart. - Beforehand, you read the release notes and compare the rendered manifests. What you do not find there should not happen in the cluster either.
helm rollbackresets the Control Plane to an earlier revision. CRDs and proxies stay as they are.
CLI 26.11.2 ─> linkerd-crds 2026.11.2 ─> linkerd-control-plane 2026.11.2
─> rollout restart (proxies 26.11.2) ─> linkerd-multicluster 2026.11.2
meanwhile: last ─> helper in tag15, every request logged
Drill 1: Starting Point and Measurement
The app from Day 14 serves as the measurement object, at a higher rate (pause 0.2 instead of 0.5 s):
mkdir -p ~/linkerd/tag15 && cd ~/linkerd/tag15
sed -e 's/tag14/tag15/g' -e 's/sleep 0.5/sleep 0.2/' ../tag14/app.yaml > app.yaml
kubectl --context west apply -f app.yaml
kubectl --context west -n tag15 rollout status deploy --timeout=180sExpected (abridged):
namespace/tag15 created
...
deployment "helper" successfully rolled out
deployment "last" successfully rolled out
Step 1 of the upgrade is the CLI. It has been in
/usr/local/bin as edge-26.11.2 since Day 1.
From now on, you use linkerd again for west
instead of ./linkerd-edge-26.11.1. What does the new CLI
see?
linkerd --context west version
linkerd --context west check 2>&1 | grep -A1 '^‼ control plane and cli'
linkerd --context west check --proxy -n tag15 2>&1 | grep -A3 '^‼ data plane is'Expected:
Client version: edge-26.11.2
Server version: edge-26.11.1
‼ control plane and cli versions match
control plane running edge-26.11.1 but cli running edge-26.11.2
‼ data plane is up-to-date
some proxies are not running the current version:
* helper-79d57fbbbb-bvxls (edge-26.11.1)
* helper-79d57fbbbb-zg98l (edge-26.11.1)
A newer CLI against an older Control Plane is allowed and intended –
the reverse is not. check only warns (‼) but
ends with √.
Save your progress? Sign up for free
Drill 2: Read the Release Notes
curl -fsSL https://api.github.com/repos/linkerd/linkerd2/releases/tags/edge-26.11.2 \
| jq -r '.published_at, .body' | grep -vE '^\* (ci|build|chore)' | head -5Expected (abridged):
2026-11-13T23:50:48Z
## What's Changed
* fix(inject): inherit beta native sidecar annotation from namespace by @spicyshar…
* proxy: v2.373.0 by @l5d-bot in https://github.com/linkerd/linkerd2/pull/15768
* proxy: v2.374.0 by @l5d-bot in https://github.com/linkerd/linkerd2/pull/15781
What you look for:
- Behavior changes that affect you. Here: the
injector now also inherits
config.beta.linkerd.io/proxy-enable-native-sidecarfrom the namespace. If you set the annotation on the namespace (Day 2 showed it on the Pod), you get different Pods after the restart. - Proxy versions: two proxy jumps are packed into one edge version.
- CRDs, values, removed features, notes such as “breaking” or “action required”. If you skip versions, you read the notes of all versions in between. The monthly “Edge Release Roundup” on the Linkerd blog summarizes them.
Then you check what changes in the charts:
diff <(helm show values linkerd-edge/linkerd-control-plane --version 2026.11.1) \
<(helm show values linkerd-edge/linkerd-control-plane --version 2026.11.2)
diff <(helm template x linkerd-edge/linkerd-crds --version 2026.11.1 \
| grep -v helm.sh/chart) \
<(helm template x linkerd-edge/linkerd-crds --version 2026.11.2 \
| grep -v helm.sh/chart) && echo crds-gleichExpected:
23c23
< linkerdVersion: edge-26.11.1
---
> linkerdVersion: edge-26.11.2
crds-gleich
Only the version changes; the CRDs are identical except for the chart label. A small step – this is exactly what upgrades should look like.
Save your progress? Sign up for free
Drill 3: Preview the Manifest
helm --kube-context west get manifest linkerd-control-plane -n linkerd > alt.yaml
helm template linkerd-control-plane linkerd-edge/linkerd-control-plane \
--version 2026.11.2 -n linkerd \
-f ../tag14/values-ha.yaml -f ../tag14/values-west.yaml > neu.yaml
diff alt.yaml neu.yaml | grep -E '^[<>] .*image:' | sort | uniq -cExpected:
1 < image: cr.l5d.io/linkerd/controller:edge-26.11.1
5 < image: cr.l5d.io/linkerd/controller:edge-26.11.1
6 < image: cr.l5d.io/linkerd/proxy:edge-26.11.1
1 > image: cr.l5d.io/linkerd/controller:edge-26.11.2
5 > image: cr.l5d.io/linkerd/controller:edge-26.11.2
6 > image: cr.l5d.io/linkerd/proxy:edge-26.11.2
New images for the controllers (including the heartbeat CronJob) and
for linkerd-init and linkerd-proxy in the
three Control Plane Deployments – both use the proxy image. The rest of
the diff consists of version labels and the webhook certificates, which
every render regenerates.
Warning: No
--reuse-values. With it, Helm keeps your old values but ignores new chart defaults. Pass the same values files with every upgrade.
Save your progress? Sign up for free
Drill 4: CRDs, Then Control Plane
Record the start time, then both releases one after the other:
date -u +%FT%TZ > start.txt
helm --kube-context west upgrade linkerd-crds linkerd-edge/linkerd-crds \
--version 2026.11.2 -n linkerd
helm --kube-context west upgrade linkerd-control-plane \
linkerd-edge/linkerd-control-plane --version 2026.11.2 -n linkerd \
-f ../tag14/values-ha.yaml -f ../tag14/values-west.yaml
kubectl --context west -n linkerd rollout status deploy --timeout=300s
kubectl --context west -n linkerd get podsExpected (abridged):
STATUS: deployed
REVISION: 2
...
STATUS: deployed
REVISION: 3
Waiting for deployment "linkerd-destination" rollout to finish: 1 old replicas are …
deployment "linkerd-destination" successfully rolled out
deployment "linkerd-identity" successfully rolled out
deployment "linkerd-proxy-injector" successfully rolled out
NAME READY STATUS RESTARTS AGE
linkerd-destination-6fccb5879-k944c 4/4 Running 0 11s
linkerd-destination-6fccb5879-lvqmp 4/4 Running 0 12s
linkerd-destination-755db578f5-62kmw 4/4 Terminating 0 4m14s
linkerd-identity-74946cc965-6sh5z 2/2 Running 0 11s
...
About 20 seconds for all three Deployments.
maxUnavailable: 1 from the HA values always keeps one
replica up. And the traffic?
sleep 20
kubectl --context west -n tag15 logs deploy/last -c last \
--since-time=$(cat start.txt) \
| awk '{print $2, $3}' | sort | uniq -cExpected:
188 /get 200
Not a single error. The proxies keep their connections and endpoints,
even while destination is switching.
Save your progress? Sign up for free
Drill 5: Catch Up the Data Plane
linkerd --context west version --short
linkerd --context west check --proxy -n tag15 2>&1 | grep -E '^‼|^\s+\*'Expected:
edge-26.11.2
edge-26.11.2
‼ data plane is up-to-date
* helper-79d57fbbbb-bvxls (edge-26.11.1)
* helper-79d57fbbbb-zg98l (edge-26.11.1)
* last-55f7c55d84-lg69g (edge-26.11.1)
‼ data plane and cli versions match
The Control Plane is new, the proxies are old. The injector only puts
the new version into new Pods. The server first, so
that last keeps measuring meanwhile:
date -u +%FT%TZ > start2.txt
kubectl --context west -n tag15 rollout restart deploy/helper
kubectl --context west -n tag15 rollout status deploy/helper
sleep 20
kubectl --context west -n tag15 logs deploy/last -c last \
--since-time=$(cat start2.txt) \
| awk '{print $2, $3}' | sort | uniq -cExpected (abridged):
deployment.apps/helper restarted
...
deployment "helper" successfully rolled out
128 /get 200
Then the client itself. Its log starts fresh afterwards:
kubectl --context west -n tag15 rollout restart deploy/last
kubectl --context west -n tag15 rollout status deploy/last
sleep 30
kubectl --context west -n tag15 get pods -o \
custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd --context west check --proxy -n tag15 2>&1 | grep -E 'data plane is|Status'Expected:
deployment.apps/last restarted
deployment "last" successfully rolled out
NAME PROXY
helper-5d777bbb87-7bwcn edge-26.11.2
helper-5d777bbb87-nn2dn edge-26.11.2
last-79f6974c-gjpnz edge-26.11.2
√ data plane is up-to-date
Status check results are √
Tip: Right after the rollout,
check --proxystill reports the oldlastPod. The curl loop does not react to SIGTERM and is only terminated after 30 seconds (terminationGracePeriodSeconds).
Save your progress? Sign up for free
Drill 6: Roll Back the Control Plane
Suppose the new version causes trouble. Helm knows every revision:
date -u +%FT%TZ > start3.txt
helm --kube-context west history linkerd-control-plane -n linkerd
helm --kube-context west rollback linkerd-control-plane 2 -n linkerd
kubectl --context west -n linkerd rollout status deploy --timeout=300s | tail -1
helm --kube-context west history linkerd-control-plane -n linkerd | tail -2
linkerd --context west version --shortExpected (abridged):
REVISION UPDATED STATUS CHART … DESCRIPTION
1 Thu Nov 19 20:04:09 2026 superseded linkerd-control-plane… Install complete
2 Thu Nov 19 20:08:41 2026 superseded linkerd-control-plane… Upgrade complete
3 Thu Nov 19 20:19:16 2026 deployed linkerd-control-plane… Upgrade complete
Rollback was a success! Happy Helming!
deployment "linkerd-proxy-injector" successfully rolled out
3 Thu Nov 19 20:19:16 2026 superseded … edge-26.11.2 Upgrade complete
4 Thu Nov 19 20:21:32 2026 deployed … edge-26.11.1 Rollback to 2
edge-26.11.2
edge-26.11.1
Revision 2 is the state before the upgrade (Day 14 after the anti-affinity fix). The rollback does not overwrite history; it creates revision 4 as a copy of 2.
sleep 10
kubectl --context west -n tag15 logs deploy/last -c last \
--since-time=$(cat start3.txt) \
| awk '{print $2, $3}' | sort | uniq -c
linkerd --context west check --proxy -n tag15 2>&1 | grep -E '^‼'Expected:
164 /get 200
‼ control plane proxies are up-to-date
‼ control plane proxies and cli versions match
Again no errors. What is interesting is what is not
reported: the app proxies now run edge-26.11.2 against a
Control Plane on edge-26.11.1. check only
compares with the CLI and the latest version, not proxy with Control
Plane. For one step this works fine; for larger gaps, you restart the
Data Plane after the rollback as well. The CRDs stay at 2026.11.2 – a
separate release, and you do not roll back CRDs lightly: older schemas
can drop fields of newer objects.
Save your progress? Sign up for free
Drill 7: Forward Again
Going forward uses the same command as in Drill 4, not
rollback 3. That way, the values file remains the single
source of truth:
date -u +%FT%TZ > start4.txt
helm --kube-context west upgrade linkerd-control-plane \
linkerd-edge/linkerd-control-plane --version 2026.11.2 -n linkerd \
-f ../tag14/values-ha.yaml -f ../tag14/values-west.yaml
kubectl --context west -n linkerd rollout status deploy --timeout=300s | tail -1
helm --kube-context west list -n linkerd
sleep 15
kubectl --context west -n tag15 logs deploy/last -c last \
--since-time=$(cat start4.txt) \
| awk '{print $2, $3}' | sort | uniq -c
linkerd --context west check --proxy -n tag15 2>&1 | tail -1Expected (abridged):
REVISION: 5
deployment "linkerd-proxy-injector" successfully rolled out
NAME NAMESPACE REVISION … CHART APP VERSION
linkerd-control-plane linkerd 5 … linkerd-control-plane-2026.11.2 edge-26.11.2
linkerd-crds linkerd 2 … linkerd-crds-2026.11.2
150 /get 200
Status check results are √
Upgrade, Data Plane restart, rollback, and another upgrade: in the
measured windows, about 630 requests, zero errors. That
is the reward for two replicas, a PDB, and
maxUnavailable: 1.
Save your progress? Sign up for free
Drill 8: The Multicluster Extension via Helm
Extensions come after the Control Plane, in the same version. The probe port gets a fixed place in the values. Day 12 showed why: the link on the other side and the probe Service must name the same port.
cat > mc-values.yaml <<'EOF'
gateway:
serviceType: LoadBalancer # k3s ServiceLB provides the VM's IP
probe:
port: 4191 # the other side probes this port (Day 12)
EOF
helm --kube-context west install linkerd-multicluster \
linkerd-edge/linkerd-multicluster --version 2026.11.2 \
-n linkerd-multicluster --create-namespace -f mc-values.yaml
kubectl --context west -n linkerd-multicluster rollout status deploy --timeout=180s
kubectl --context west get ns linkerd-multicluster --show-labels
kubectl --context west -n linkerd-multicluster get svc linkerd-gatewayExpected (abridged):
STATUS: deployed
REVISION: 1
deployment "linkerd-gateway" successfully rolled out
deployment "linkerd-local-service-mirror" successfully rolled out
NAME STATUS AGE LABELS
linkerd-multicluster Active 11s …,linkerd.io/extension=multicluster,…
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
linkerd-gateway LoadBalancer 10.43.97.226 172.16.0.182 4143:31923/TCP,4191:32…
Here --create-namespace is enough: a Job of the chart
(hook post-install) sets the label
linkerd.io/extension=multicluster, by which
linkerd check recognizes the extension. The Link CRD is in
the same chart. For ost to read west as on Day
12, nothing else is needed on west. If west is
to read ost later, you add an entry under
controllers with gateway.probe.port: 30980
(NodePort on ost) – before the link.
linkerd --context west multicluster check 2>&1 | tail -5
linkerd --context west check 2>&1 | tail -1Expected:
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match
Status check results are √
Status check results are √
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the proxy version for every Pod in
tag15. Target time: 30 s - Find the chart version for
edge-26.10.2. Target time: 20 s - Show all Pods on
westwhose proxy is notedge-26.11.2. Target time: 60 s - Show which revision of the Control Plane is currently active. Target time: 15 s
Solution
kubectl --context west -n tag15 get pods -o \
custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
helm search repo linkerd-edge/linkerd-control-plane --versions | grep edge-26.10.2
kubectl --context west get pods -A -l linkerd.io/control-plane-ns=linkerd -o \
custom-columns=NS:.metadata.namespace,NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version | grep -v edge-26.11.2
helm --kube-context west history linkerd-control-plane -n linkerd | grep deployedExpected (abridged, task 3 during the Challenge):
NAME PROXY
helper-5d777bbb87-7bwcn edge-26.11.2
...
linkerd-edge/linkerd-control-plane 2026.10.2 edge-26.10.2 Linkerd gives you…
NS NAME PROXY
tag15 last-748d866d8c-8h9px edge-26.11.1
5 Thu Nov 19 20:22:22 2026 deployed linkerd-control-plane-2026.11.2 …
Save your progress? Sign up for free
Checkpoint
helm --kube-context west list -A -o json | jq -r '.[] | .name + " " + .chart'Expected:
gateway-api-crd gateway-api-crd-1.6.301
linkerd-control-plane linkerd-control-plane-2026.11.2
linkerd-crds linkerd-crds-2026.11.2
linkerd-multicluster linkerd-multicluster-2026.11.2
linkerd --context west check 2>&1 | tail -1
linkerd --context west multicluster check 2>&1 | tail -1Expected: Status check results are √
twice.
kubectl --context west get nsExpected: after cleanup, no tag14 and
no tag15, but linkerd and
linkerd-multicluster.
Challenge
You want to see the new proxy version on just one workload first – or
keep a workload on the old one temporarily. Pin last to
edge-26.11.1, show the mixed Data Plane, and remove the pin
again.
Hint: The proxy version can be chosen per Pod template via an annotation.
Solution
kubectl --context west -n tag15 patch deploy last -p \
'{"spec":{"template":{"metadata":{"annotations":{
"config.linkerd.io/proxy-version":"edge-26.11.1"}}}}}'
kubectl --context west -n tag15 rollout status deploy/last
sleep 40
kubectl --context west -n tag15 get pods -o \
custom-columns=NAME:.metadata.name,\
PROXY:.metadata.annotations.linkerd\\.io/proxy-version
kubectl --context west -n tag15 logs deploy/last -c last --since=30s \
| awk '{print $2, $3}' | sort | uniq -cExpected:
deployment.apps/last patched
deployment "last" successfully rolled out
NAME PROXY
helper-5d777bbb87-7bwcn edge-26.11.2
helper-5d777bbb87-nn2dn edge-26.11.2
last-748d866d8c-8h9px edge-26.11.1
125 /get 200
Old client, new server, no errors. Remove the pin (~1
stands for / in the JSON path):
kubectl --context west -n tag15 patch deploy last --type=json -p '[{"op":"remove",
"path":"/spec/template/metadata/annotations/config.linkerd.io~1proxy-version"}]'
kubectl --context west -n tag15 rollout status deploy/lastExpected:
deployment.apps/last patched
deployment "last" successfully rolled out
The same principle underpins the production strategy: after upgrading the Control Plane, you first restart a canary namespace and watch it before the rest follows. The annotation on the namespace keeps teams that are not ready yet on the old version for a while.
Save your progress? Sign up for free
Cleanup
kubectl --context west delete namespace tag15
rm -f ../tag14/linkerd-edge-26.11.1 alt.yaml neu.yaml start*.txt
kubectl --context west get nsExpected:
namespace "tag15" deleted
NAME STATUS AGE
default Active 5h4m
kube-node-lease Active 5h4m
kube-public Active 5h4m
kube-system Active 5h4m
linkerd Active 22m
linkerd-multicluster Active 71s
west now runs permanently via Helm with
edge-26.11.2. For a new link ost → west, you
install the controller on ost as on Day 12, Drill 5, but
with ../tag12/mc-ost-gw.yaml (otherwise the NodePort
gateway disappears), and create the link as in Drill 6 –
link-gen still runs via CLI, unchanged. From now on, you
change the extension on west only via
helm upgrade with mc-values.yaml.
Interview Note
- In what order do you upgrade Linkerd? CLI, CRDs, Control Plane, then the Data Plane via restart, and finally the extensions. Release notes and diff beforehand.
- What does
helm rollbackdo – and what does it not do? It applies an earlier revision of the Control Plane as a new revision. CRDs (a separate release) and running proxies stay unchanged. - How do you notice that proxies are outdated?
linkerd check --proxyand the annotationlinkerd.io/proxy-versionon every Pod.
Enterprise Note
- Keep up with edge regularly, in small steps. Vendors’ stable lines only minor by minor (2.19 → 2.20); when skipping edge versions, read all release notes.
- First a staging environment, then in production a canary namespace with a load generator and an SLO dashboard, and only then the rest of the Data Plane.
- Do not leave proxies old across several upgrades. Testing mainly covers a new Control Plane with proxies of the immediately preceding version.
- Stagger Data Plane restarts (namespace by namespace) and keep the apps’ PDBs in mind: single replicas such as databases go down briefly with every upgrade.
- Upgrades via GitOps: bumping the chart version in Git is the change,
reverting it is the rollback. Otherwise, a manual
helm rollbackcreates drift from the Git state.