Day 14: Three-Way Merge, Server-Side Apply and Helm 4
Goal: Understand how Helm 3 and Helm 4 deal with manual changes and who owns a field, and move a Helm 3 release to Helm 4 without surprises. · Duration: approx. 75 min · Prerequisite: Day 13
Briefing
- Helm 3 patches on the client side with a three-way merge of old manifest, live object and new manifest. Drift in fields the chart renders is silently overwritten.
- Helm 4 installs new releases with Server-Side Apply (SSA). Every field has a field manager; if Helm wants to change a field someone else owns, you get a conflict.
- A release keeps its method (
--server-side=auto). Helm 3 releases stay client-side until you switch them. - Helm 3 gets security updates until 10 February 2027. Both versions read the same release Secrets.
- In Helm 4, plugins have types (
cli/v1,getter/v1,postrenderer/v1) and their signatures are verified.
Drill 1: Install Helm 3 alongside
For the comparison, Helm 3.22.1 goes next to Helm 4 as
helm3 – with a checksum, as on Day 0:
# on prod
mkdir -p ~/helm/tag14 && cd ~/helm/tag14
ARCH=$(dpkg --print-architecture)
curl -fsSLO https://get.helm.sh/helm-v3.22.1-linux-$ARCH.tar.gz
curl -fsSLO https://get.helm.sh/helm-v3.22.1-linux-$ARCH.tar.gz.sha256sum
sha256sum -c helm-v3.22.1-linux-$ARCH.tar.gz.sha256sum
tar -xzf helm-v3.22.1-linux-$ARCH.tar.gz --strip-components=1 linux-$ARCH/helm
sudo install -m 755 helm /usr/local/bin/helm3 && rm helm
helm3 version --short
helm version --short
helm3 list -A -q
diff <(helm env HELM_PLUGINS) <(helm3 env HELM_PLUGINS) && echo "gleicher Plugin-Ordner"Expected:
helm-v3.22.1-linux-amd64.tar.gz: OK
v3.22.1+gd6c1a0e
v4.3.2+g8346aaa
gateway-api-crd
helper
notes
notes-crds
traefik
traefik-crd
zot
gleicher Plugin-Ordner
Helm 3 sees every release Helm 4 created: the storage format (Secrets
sh.helm.release.v1.*) is the same.
gleicher Plugin-Ordner means “same plugin directory”: both
also share repositories, registry login and the plugin directory – that
becomes important in Drill 7.
Save your progress? Sign up for free
Drill 2: Drift under Helm 3 – the three-way merge
A small throwaway chart, so that notes stays
untouched:
# on prod
cd ~/helm/tag14 && mkdir -p drift/templates
cat > drift/Chart.yaml <<'EOF'
# drift/Chart.yaml – Stand Tag 14
apiVersion: v2
name: drift
version: 0.1.0
appVersion: "2.26.0"
EOF
cat > drift/values.yaml <<'EOF'
# drift/values.yaml – Stand Tag 14
replicaCount: 2
logLevel: info
EOF
cat > drift/templates/deployment.yaml <<'EOF'
{{/* drift/templates/deployment.yaml – Stand Tag 14 */}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-drift
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
containers:
- name: httpbin
image: ghcr.io/mccutchen/go-httpbin:{{ .Chart.AppVersion }}
env:
- name: LOG_LEVEL
value: {{ .Values.logLevel | quote }}
resources:
requests:
cpu: 10m
memory: 16Mi
EOFInstall with Helm 3, then “fix it by hand” at night, and upgrade without any change:
# on prod
helm3 install d3 ./drift -n tag14 --create-namespace --wait | grep STATUS
kubectl -n tag14 scale deploy d3-drift --replicas=4
kubectl -n tag14 set env deploy/d3-drift LOG_LEVEL=debug EXTRA=hand
kubectl -n tag14 label deploy d3-drift notfall=ja
helm3 upgrade d3 ./drift -n tag14 --wait | grep REVISION
F='{.spec.replicas} {.metadata.labels.notfall} '
F+='{range .spec.template.spec.containers[0].env[*]}{.name}={.value} {end}{"\n"}'
kubectl -n tag14 get deploy d3-drift -o jsonpath="$F"Expected:
STATUS: deployed
deployment.apps/d3-drift scaled
deployment.apps/d3-drift env updated
deployment.apps/d3-drift labeled
REVISION: 2
2 ja LOG_LEVEL=info EXTRA=hand
The label notfall=ja means “emergency=yes”.
| Field | old / new (chart) | live | after helm3 upgrade |
|---|---|---|---|
replicas |
2 / 2 | 4 | 2 – drift overwritten |
LOG_LEVEL |
info / info | debug | info – drift overwritten |
EXTRA |
– / – | hand | hand – stays |
Label notfall |
– / – | ja | ja – stays |
Helm 3 compares the new manifest with the live object and resets everything the chart renders. Whatever the chart does not know about stays. The night’s scaling is gone – without a warning.
Save your progress? Sign up for free
Drill 3: The same drift under Helm 4
# on prod
helm install d4 ./drift -n tag14 --wait | grep STATUS
kubectl -n tag14 scale deploy d4-drift --replicas=4
kubectl -n tag14 set env deploy/d4-drift LOG_LEVEL=debug EXTRA=hand
kubectl -n tag14 label deploy d4-drift notfall=ja
helm upgrade d4 ./drift -n tag14 --wait
kubectl -n tag14 get deploy d4-drift -o jsonpath="$F"Expected (abridged):
STATUS: deployed
deployment.apps/d4-drift scaled
deployment.apps/d4-drift env updated
deployment.apps/d4-drift labeled
Error: UPGRADE FAILED: conflict occurred while applying object tag14/d4-drift apps/v1, K
ind=Deployment: Apply failed with 2 conflicts: conflicts with "kubectl" with subresource
"scale" using apps/v1:
- .spec.replicas
conflicts with "kubectl-set" using apps/v1:
- .spec.template.spec.containers[name="httpbin"].env[name="LOG_LEVEL"].value
4 ja LOG_LEVEL=debug EXTRA=hand
Before the error, Helm 4 prints the same message once more as
level=WARN msg="upgrade failed". Instead of silently
overwriting, SSA refuses the upgrade and names the field and the field
manager. Who owns what is in managedFields:
# on prod
kubectl -n tag14 get deploy d4-drift --show-managed-fields -o json \
| jq -r '.metadata.managedFields[]
| [.manager, .operation, .subresource // "-"] | @tsv'
helm upgrade d4 ./drift -n tag14 --force-conflicts --wait | grep REVISION
kubectl -n tag14 get deploy d4-drift -o jsonpath="$F"
helm history d4 -n tag14 -o json | jq -r '.[] | "\(.revision) \(.status)"'Expected:
helm Apply -
kubectl Update scale
kubectl-set Update -
kubectl-label Update -
k3s Update status
REVISION: 3
2 ja LOG_LEVEL=info EXTRA=hand
1 superseded
2 failed
3 deployed
Apply is SSA, Update a classic write.
kubectl scale writes through the scale
subresource, kubectl set env is called
kubectl-set. --force-conflicts takes the
contested fields back; EXTRA and the label still belong to
their managers and stay. The result matches Helm 3 – but only after
someone has decided on purpose. Day 19 shows the same conflict in
operations.
Save your progress? Sign up for free
Drill 4: A Helm 3 release under Helm 4
# on prod
helm get metadata d3 -n tag14 | grep APPLY_METHOD
helm get metadata d4 -n tag14 | grep APPLY_METHOD
kubectl -n tag14 scale deploy d3-drift --replicas=3
helm upgrade d3 ./drift -n tag14 --wait | grep REVISION
helm get metadata d3 -n tag14 | grep APPLY_METHOD
kubectl -n tag14 get deploy d3-drift -o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n tag14 get deploy d3-drift --show-managed-fields -o json \
| jq -r '.metadata.managedFields[] | select(.manager=="helm") | .operation'Expected:
APPLY_METHOD: client-side apply (defaulted)
APPLY_METHOD: server-side apply
deployment.apps/d3-drift scaled
REVISION: 3
APPLY_METHOD: client-side apply
2
Update
Helm 3 stores no method – Helm 4 then assumes client-side
(“defaulted”). With the default --server-side=auto, Helm 4
sticks with that and behaves like Helm 3: the drift is overwritten
without a conflict. Switching is one flag:
# on prod
helm upgrade d3 ./drift -n tag14 --server-side=true --wait | grep REVISION
helm get metadata d3 -n tag14 | grep APPLY_METHOD
kubectl -n tag14 get deploy d3-drift --show-managed-fields -o json \
| jq -r '.metadata.managedFields[] | select(.manager=="helm") | .operation'Expected:
REVISION: 4
APPLY_METHOD: server-side apply
Apply
Helm has converted its old Update entry into an
Apply entry; from now on auto means
server-side. Be careful during the transition if someone still uses Helm
3:
# on prod
helm3 upgrade d4 ./drift -n tag14 --wait | grep REVISION
helm get metadata d4 -n tag14 | grep APPLY_METHOD
helm3 rollback d4 3 -n tag14 | tail -1
helm get metadata d4 -n tag14 | grep APPLY_METHOD
helm upgrade d4 ./drift -n tag14 --server-side=true --wait | grep REVISIONExpected:
REVISION: 4
APPLY_METHOD: client-side apply (defaulted)
Rollback was a success! Happy Helming!
APPLY_METHOD: client-side apply (defaulted)
REVISION: 6
A single helm3 upgrade silently switches an SSA release
back to client-side – and so does a Helm 3 rollback to an SSA revision.
So: first move all pipelines and admin machines to Helm 4, then
switch.
Save your progress? Sign up for free
Drill 5: Field managers and the autoscaler
The usual chart pattern: if an HPA is enabled, the chart does not
render replicas. Version 0.2.0 of drift:
# on prod
sed -i 's/^version: 0.1.0/version: 0.2.0/' drift/Chart.yaml
cat > drift/values.yaml <<'EOF'
# drift/values.yaml – Stand Tag 14
replicaCount: 2
logLevel: info
autoscaling:
enabled: false
minReplicas: 3
maxReplicas: 5
EOF
cat > drift/templates/deployment.yaml <<'EOF'
{{/* drift/templates/deployment.yaml – Stand Tag 14 */}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-drift
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
containers:
- name: httpbin
image: ghcr.io/mccutchen/go-httpbin:{{ .Chart.AppVersion }}
env:
- name: LOG_LEVEL
value: {{ .Values.logLevel | quote }}
resources:
requests:
cpu: 10m
memory: 16Mi
EOF
cat > drift/templates/hpa.yaml <<'EOF'
{{/* drift/templates/hpa.yaml – Stand Tag 14 */}}
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ .Release.Name }}-drift
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ .Release.Name }}-drift
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 80
{{- end }}
EOF
helm template d3 ./drift -s templates/deployment.yaml --set autoscaling.enabled=true \
| sed -n 3,8pExpected:
apiVersion: apps/v1
kind: Deployment
metadata:
name: d3-drift
spec:
selector:
Now switch a running release to the HPA:
# on prod
helm install h1 ./drift -n tag14 --wait | grep STATUS
helm upgrade h1 ./drift -n tag14 --set autoscaling.enabled=true | grep REVISION
sleep 20
kubectl -n tag14 get events -o custom-columns=MELDUNG:.message \
--field-selector involvedObject.kind=Deployment,involvedObject.name=h1-driftExpected:
STATUS: deployed
REVISION: 2
MELDUNG
Scaled up replica set h1-drift-654cfb8d45 from 0 to 2
Scaled down replica set h1-drift-654cfb8d45 from 2 to 1
Scaled up replica set h1-drift-654cfb8d45 from 1 to 3
The column header MELDUNG means “message”. The trap:
replicas belonged to Helm alone. When Helm stops rendering
the field, SSA removes it – Kubernetes sets the default 1, and only then
does the HPA scale back up. Under load, pods are missing at that moment.
(The lab has no metrics-server; the HPA can still raise the count to
minReplicas.) It works cleanly if the field changes owner
first:
# on prod
helm install h2 ./drift -n tag14 --wait | grep STATUS
kubectl -n tag14 scale deploy h2-drift --replicas=3
helm upgrade h2 ./drift -n tag14 --set autoscaling.enabled=true | grep REVISION
sleep 20
kubectl -n tag14 get events -o custom-columns=MELDUNG:.message \
--field-selector involvedObject.kind=Deployment,involvedObject.name=h2-drift
kubectl -n tag14 get hpa h2-driftExpected (abridged):
STATUS: deployed
deployment.apps/h2-drift scaled
REVISION: 2
MELDUNG
Scaled up replica set h2-drift-7bd5c8d666 from 0 to 2
Scaled up replica set h2-drift-7bd5c8d666 from 2 to 3
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS ...
h2-drift Deployment/h2-drift cpu: <unknown>/80% 3 5 3 ...
kubectl scale takes over spec.replicas;
during the upgrade Helm only gives up what it no longer owns – nothing
drops to 1. The same principle applies to any other actor: if something
manages a field permanently (HPA, VPA, an operator), the chart must not
render it.
Save your progress? Sign up for free
Drill 6: What else changes
# on prod
helm upgrade d4 ./drift -n tag14 --atomic --force --dry-run=client 2>&1 | head -2
helm install weg ./drift -n tag14 --set replicaCount=0 | grep STATUS
helm uninstall weg -n tag14 --keep-history
helm list -n tag14 -q | tr '\n' ' '; echo
helm3 list -n tag14 -q | tr '\n' ' '; echo
helm3 list -n tag14 -q -a | tr '\n' ' '; echo
mkdir -p dur/templates && printf 'apiVersion: v2\nname: dur\nversion: 0.1.0\n' \
> dur/Chart.yaml
cat > dur/templates/cm.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: dur
data:
sekunden: {{ durationSeconds "2m" | quote }}
EOF
helm template dur ./dur | tail -1
helm3 template dur ./dur 2>&1 | head -1Expected:
Flag --atomic has been deprecated, use --rollback-on-failure instead
Flag --force has been deprecated, use --force-replace instead
STATUS: deployed
release "weg" uninstalled
d3 d4 h1 h2 weg
d3 d4 h1 h2
d3 d4 h1 h2 weg
sekunden: "120"
Error: parse error at (dur/templates/cm.yaml:6): function "durationSeconds" not defined
The release weg (“gone”) is uninstalled but keeps its
history; the key sekunden means “seconds”.
| Helm 3 | Helm 4 | Day |
|---|---|---|
--atomic, --force |
--rollback-on-failure,
--force-replace |
13, 19 |
--wait = everything ready |
--wait=watcher (kstatus), default
hookOnly |
13 |
list only deployed/failed, -a for all |
list shows every status |
1 |
| post-renderer = program | post-renderer = plugin | 14 |
| client-side patch | SSA for new releases | 14 |
Charts that use the duration functions (new in 4.3, Day 7) no longer
render under Helm 3. A chart that has to serve both versions does
without them, or documents the minimum version with an annotation in
Chart.yaml.
Save your progress? Sign up for free
Drill 7: A post-renderer as a plugin
A post-renderer receives the rendered YAML on stdin and prints it
modified – here Kustomize (built into kubectl) puts a label
team on everything:
# on prod
cd ~/helm/tag14 && mkdir -p team-label
cat > team-label/render.sh <<'EOF'
#!/bin/sh
# team-label/render.sh – Stand Tag 14
set -e
dir=$(mktemp -d)
cat > "$dir/all.yaml"
cat > "$dir/kustomization.yaml" <<'KEOF'
resources: [all.yaml]
labels:
- pairs: {team: plattform}
includeSelectors: false
KEOF
kubectl kustomize "$dir"
rm -rf "$dir"
EOF
chmod +x team-label/render.sh
helm3 template d4 ./drift --post-renderer ./team-label/render.sh \
| grep -A1 'labels:' | head -2
helm template d4 ./drift --post-renderer ./team-label/render.sh 2>&1 | fold -w 88Expected:
labels:
team: plattform
Error: invalid argument "./team-label/render.sh" for "--post-renderer" flag: plugin: {Na
me:./team-label/render.sh Type:postrenderer/v1} not found
Helm 3 takes a path, Helm 4 only the name of a plugin of type
postrenderer/v1. The plugin is a plugin.yaml
next to the program:
# on prod
cat > team-label/plugin.yaml <<'EOF'
# team-label/plugin.yaml – Stand Tag 14
apiVersion: v1
type: postrenderer/v1
name: team-label
version: 0.1.0
runtime: subprocess
runtimeConfig:
platformCommand:
- command: ${HELM_PLUGIN_DIR}/render.sh
EOF
helm plugin install ./team-label
helm plugin list
helm upgrade d4 ./drift -n tag14 --post-renderer team-label --wait | grep REVISION
kubectl -n tag14 get deploy -l team=plattform
helm upgrade d4 ./drift -n tag14 --wait | grep REVISION
kubectl -n tag14 get deploy -l team=plattform
helm3 list -n tag14 -q 2>&1 | head -1 | fold -w 88Expected:
Installing plugin from local directory (development mode)
Installed plugin: team-label
NAME VERSION TYPE APIVERSION PROVENANCE SOURCE
team-label 0.1.0 postrenderer/v1 v1 local dev unknown
REVISION: 7
NAME READY UP-TO-DATE AVAILABLE AGE
d4-drift 2/2 2 2 4m18s
REVISION: 8
No resources found in tag14 namespace.
failed to load plugins: failed to load plugin at "/home/ubuntu/.local/share/helm/plugins
/team-label/plugin.yaml": error unmarshaling JSON: while decoding JSON: json: unknown fi
eld "apiVersion"
Three lessons: a local directory counts as “local dev” and needs no
signature. Helm does not store the post-renderer in the
release – the next upgrade without the flag removes the label again. And
because both versions use the same plugin directory, Helm 3 complains
about every plugin in the new format (apiVersion: v1).
Besides subprocesses, Helm 4 also knows WebAssembly plugins
(runtime: extism/v1) – not needed in the lab.
Save your progress? Sign up for free
Drill 8: Install helm-diff with signature check
By default, Helm 4 verifies plugin tarballs against a
.prov file (Day 16 shows the same for charts). helm-diff
ships both:
# on prod
mkdir -p ~/helm/tag14/diff && cd ~/helm/tag14/diff
ARCH=$(dpkg --print-architecture)
U=https://github.com/databus23/helm-diff/releases/download/v3.15.15
for f in diff-3.15.15-linux-$ARCH.tgz diff-3.15.15-linux-$ARCH.tgz.prov \
helm-diff_3.15.15_checksums.txt pubkey.asc; do curl -fsSLO $U/$f; done
grep " diff-3.15.15-linux-$ARCH.tgz$" helm-diff_3.15.15_checksums.txt | sha256sum -c
helm plugin install diff-3.15.15-linux-$ARCH.tgz 2>&1 | tail -1 | fold -w 88
gpg --show-keys pubkey.asc | sed -n 2p
gpg --no-default-keyring --keyring ./helm-diff.kbx --import pubkey.asc 2>&1 | tail -1
helm plugin verify diff-3.15.15-linux-$ARCH.tgz --keyring ./helm-diff.kbx
helm plugin install diff-3.15.15-linux-$ARCH.tgz --keyring ./helm-diff.kbx | tail -1
helm plugin list | cut -c1-70Expected (abridged):
diff-3.15.15-linux-amd64.tgz: OK
Error: plugin verification failed: reading keybox /home/ubuntu/.gnupg/pubring.kbx: keybo
x contains no OpenPGP keys
C5645EF47482257A1F806D2BEA17A2A206AFF8CD
gpg: imported: 1
Signed by: Fabian Ruff (helm-diff release key) <…>
Using Key With Fingerprint: C5645EF47482257A1F806D2BEA17A2A206AFF8CD
Plugin Hash Verified: sha256:504e9941…
Installed plugin: diff
NAME VERSION TYPE APIVERSION PROVENANCE SO
diff 3.15.15 cli/v1 legacy mismatched provenance un
team-label 0.1.0 postrenderer/v1 v1 local dev un
Without a key the installation aborts – good. With a dedicated
keyring that contains only the release key, it works. Compare the
fingerprint with the project page. “mismatched provenance” in the list
is cosmetic: the .prov names
diff-3.15.15-linux-amd64.tgz, Helm stores the copy as
diff-3.15.15.tgz. legacy means: old plugin
format, runs under both versions. Now the payoff:
# on prod
cd ~/helm/tag14
helm diff upgrade d4 ./drift -n tag14 --set logLevel=debug -C 1
kubectl -n tag14 set env deploy/d4-drift LOG_LEVEL=warn
helm diff upgrade d4 ./drift -n tag14 -C 1 --detailed-exitcode; echo "Exit: $?"
helm diff upgrade d4 ./drift -n tag14 -C 1 --three-way-merge --detailed-exitcode
echo "Exit: $?"
helm upgrade d4 ./drift -n tag14 2>&1 | grep '^Error' \
| grep -o 'conflict with "[a-z-]*"'
helm upgrade d4 ./drift -n tag14 --force-conflicts --wait | grep STATUSExpected (abridged):
tag14, d4-drift, Deployment (apps) has changed:
...
- name: LOG_LEVEL
- value: "info"
+ value: "debug"
...
deployment.apps/d4-drift env updated
Exit: 0
tag14, d4-drift, Deployment (apps) has changed:
...
- name: LOG_LEVEL
- value: warn
+ value: info
...
Error: identified at least one change, exiting with non-zero exit code (…)
Error: plugin "diff" exited with error
Exit: 2
conflict with "kubectl-set"
STATUS: deployed
Without --three-way-merge, helm-diff compares the stored
manifest with the new one – the drift stays invisible (exit 0). With the
flag it compares against the live object and shows exactly the field the
SSA upgrade then fails on. --detailed-exitcode (2 =
changes) turns this into a pipeline stage.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the apply method for all releases in
tag14. Target time: 30 s - Who owns
spec.replicasofh2-drift? Target time: 30 s - List the installed plugins with their type. Target time: 15 s
- Does
driftwith HPA render the same under Helm 3 and Helm 4? Ignore blank lines. Target time: 30 s
Solution
# on prod
for r in d3 d4 h1 h2; do echo "$r $(helm get metadata $r -n tag14 | grep APPLY_METHOD)"
done
kubectl -n tag14 get deploy h2-drift --show-managed-fields -o json \
| jq -r '.metadata.managedFields[]
| select(.fieldsV1."f:spec"."f:replicas") | .manager'
helm plugin list | awk 'NR>1 {print $1, $3}'
diff -B <(helm template x ./drift --set autoscaling.enabled=true) \
<(helm3 template x ./drift --set autoscaling.enabled=true 2>/dev/null) \
&& echo identischExpected:
d3 APPLY_METHOD: server-side apply
d4 APPLY_METHOD: server-side apply
h1 APPLY_METHOD: server-side apply
h2 APPLY_METHOD: server-side apply
kubectl
diff cli/v1
team-label postrenderer/v1
identisch
identisch means “identical”. Without -B,
diff reports one line: Helm 4 puts a blank line before
every ---. If you compare renderings from different Helm
versions in CI, this will trip you up.
Save your progress? Sign up for free
Checkpoint
# on prod
helm3 version --short 2>/dev/nullExpected: v3.22.1+gd6c1a0e
# on prod
helm get metadata d3 -n tag14 | grep APPLY_METHODExpected:
APPLY_METHOD: server-side apply – the Helm 3 release has
been switched.
# on prod
helm diff versionExpected: 3.15.15
# on prod
helm status notes -n notes -o json | jq -r .info.statusExpected: deployed – notes
was not touched today.
Challenge
Install c with Helm 3, scale it to 3 by hand and switch
it to Server-Side Apply with Helm 4. Beforehand, it must be visible what
changes, and the history must not contain a failed attempt.
Hint: helm-diff with --three-way-merge
shows the drift; one upgrade can carry --server-side=true
and --force-conflicts at the same time.
Solution
# on prod
helm3 install c ./drift -n tag14 --wait 2>/dev/null | grep STATUS
kubectl -n tag14 scale deploy c-drift --replicas=3
helm diff upgrade c ./drift -n tag14 --three-way-merge -C 0 | grep '^[-+]'
helm upgrade c ./drift -n tag14 --server-side=true --force-conflicts --wait \
| grep REVISION
helm get metadata c -n tag14 | grep APPLY_METHOD
kubectl -n tag14 get deploy c-drift -o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n tag14 get deploy c-drift --show-managed-fields -o json \
| jq -r '.metadata.managedFields[] | [.manager, .operation] | @tsv'
helm history c -n tag14 -o json | jq -r '.[] | "\(.revision) \(.status)"'Expected:
STATUS: deployed
deployment.apps/c-drift scaled
- replicas: 3
+ replicas: 2
REVISION: 2
APPLY_METHOD: server-side apply
2
helm Apply
k3s Update
1 superseded
2 deployed
2>/dev/null hides the Helm 3 plugin warning from
Drill 7. Without --force-conflicts, the first SSA upgrade
would have failed on the manager kubectl – and would still
have switched the method to server-side. The diff beforehand is the
conscious decision: the night’s 3 replicas disappear.
Save your progress? Sign up for free
Cleanup
# on prod
helm uninstall d3 d4 h1 h2 c weg -n tag14
helm plugin uninstall team-label
kubectl delete namespace tag14Expected:
release "d3" uninstalled
release "d4" uninstalled
release "h1" uninstalled
release "h2" uninstalled
release "c" uninstalled
release "weg" uninstalled
Uninstalled plugin: team-label
namespace "tag14" deleted
helm3 and helm-diff stay installed. Nothing has changed
in the chart notes; there is no commit.
Interview Note
- What happens to a manual change on the next
helm upgrade? Helm 3: fields the chart renders are silently reset (three-way merge), foreign fields stay. Helm 4 with SSA: a conflict naming the field and the field manager; only--force-conflictstakes over. - How do you migrate from Helm 3 to Helm 4? First move all
clients to Helm 4 (Helm 3 switches SSA releases back), check plugins for
the new format, make drift visible with
helm diff --three-way-merge, then run--server-side=trueonce per release. - Why does a chart with an HPA not render
replicas? Otherwise Helm and the HPA fight over the field. When switching, the field has to change owner, or it briefly drops to 1.
Enterprise Note
- Pin the Helm version centrally in pipelines, admin images and GitOps tools; one forgotten Helm 3 runner silently undoes the switch.
- SSA conflicts are a signal, not a disruption: instead of adding
--force-conflictsto every pipeline across the board, build in drift checks (helm-diff, GitOps drift detection) and clarify who should really manage the field. - Install plugins only with a verified signature, keep the keys in a
separate keyring per plugin, document the fingerprints.
--verify=falsedoes not belong in automation. - Post-renderers are invisible to
helm get– the result belongs in the review (helm get manifestafter the deployment), the plugin in every pipeline. - Prefer installing charts by digest rather than by version once they live in a registry (Day 15).