Skip to content
Hands-on IT IT Drills

Day 14: Three-Way Merge, Server-Side Apply and Helm 4

Goal: Understand how Helm 3 and Helm 4 deal with manual changes and who owns a field, and move a Helm 3 release to Helm 4 without surprises. · Duration: approx. 75 min · Prerequisite: Day 13

Briefing

  • Helm 3 patches on the client side with a three-way merge of old manifest, live object and new manifest. Drift in fields the chart renders is silently overwritten.
  • Helm 4 installs new releases with Server-Side Apply (SSA). Every field has a field manager; if Helm wants to change a field someone else owns, you get a conflict.
  • A release keeps its method (--server-side=auto). Helm 3 releases stay client-side until you switch them.
  • Helm 3 gets security updates until 10 February 2027. Both versions read the same release Secrets.
  • In Helm 4, plugins have types (cli/v1, getter/v1, postrenderer/v1) and their signatures are verified.

Drill 1: Install Helm 3 alongside

For the comparison, Helm 3.22.1 goes next to Helm 4 as helm3 – with a checksum, as on Day 0:

# on prod
mkdir -p ~/helm/tag14 && cd ~/helm/tag14
ARCH=$(dpkg --print-architecture)
curl -fsSLO https://get.helm.sh/helm-v3.22.1-linux-$ARCH.tar.gz
curl -fsSLO https://get.helm.sh/helm-v3.22.1-linux-$ARCH.tar.gz.sha256sum
sha256sum -c helm-v3.22.1-linux-$ARCH.tar.gz.sha256sum
tar -xzf helm-v3.22.1-linux-$ARCH.tar.gz --strip-components=1 linux-$ARCH/helm
sudo install -m 755 helm /usr/local/bin/helm3 && rm helm
helm3 version --short
helm version --short
helm3 list -A -q
diff <(helm env HELM_PLUGINS) <(helm3 env HELM_PLUGINS) && echo "gleicher Plugin-Ordner"

Expected:

helm-v3.22.1-linux-amd64.tar.gz: OK
v3.22.1+gd6c1a0e
v4.3.2+g8346aaa
gateway-api-crd
helper
notes
notes-crds
traefik
traefik-crd
zot
gleicher Plugin-Ordner

Helm 3 sees every release Helm 4 created: the storage format (Secrets sh.helm.release.v1.*) is the same. gleicher Plugin-Ordner means “same plugin directory”: both also share repositories, registry login and the plugin directory – that becomes important in Drill 7.

Save your progress? Sign up for free

Drill 2: Drift under Helm 3 – the three-way merge

A small throwaway chart, so that notes stays untouched:

# on prod
cd ~/helm/tag14 && mkdir -p drift/templates
cat > drift/Chart.yaml <<'EOF'
# drift/Chart.yaml – Stand Tag 14
apiVersion: v2
name: drift
version: 0.1.0
appVersion: "2.26.0"
EOF
cat > drift/values.yaml <<'EOF'
# drift/values.yaml – Stand Tag 14
replicaCount: 2
logLevel: info
EOF
cat > drift/templates/deployment.yaml <<'EOF'
{{/* drift/templates/deployment.yaml – Stand Tag 14 */}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-drift
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
      containers:
        - name: httpbin
          image: ghcr.io/mccutchen/go-httpbin:{{ .Chart.AppVersion }}
          env:
            - name: LOG_LEVEL
              value: {{ .Values.logLevel | quote }}
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
EOF

Install with Helm 3, then “fix it by hand” at night, and upgrade without any change:

# on prod
helm3 install d3 ./drift -n tag14 --create-namespace --wait | grep STATUS
kubectl -n tag14 scale deploy d3-drift --replicas=4
kubectl -n tag14 set env deploy/d3-drift LOG_LEVEL=debug EXTRA=hand
kubectl -n tag14 label deploy d3-drift notfall=ja
helm3 upgrade d3 ./drift -n tag14 --wait | grep REVISION
F='{.spec.replicas} {.metadata.labels.notfall} '
F+='{range .spec.template.spec.containers[0].env[*]}{.name}={.value} {end}{"\n"}'
kubectl -n tag14 get deploy d3-drift -o jsonpath="$F"

Expected:

STATUS: deployed
deployment.apps/d3-drift scaled
deployment.apps/d3-drift env updated
deployment.apps/d3-drift labeled
REVISION: 2
2 ja LOG_LEVEL=info EXTRA=hand

The label notfall=ja means “emergency=yes”.

Field old / new (chart) live after helm3 upgrade
replicas 2 / 2 4 2 – drift overwritten
LOG_LEVEL info / info debug info – drift overwritten
EXTRA – / – hand hand – stays
Label notfall – / – ja ja – stays

Helm 3 compares the new manifest with the live object and resets everything the chart renders. Whatever the chart does not know about stays. The night’s scaling is gone – without a warning.

Save your progress? Sign up for free

Drill 3: The same drift under Helm 4

# on prod
helm install d4 ./drift -n tag14 --wait | grep STATUS
kubectl -n tag14 scale deploy d4-drift --replicas=4
kubectl -n tag14 set env deploy/d4-drift LOG_LEVEL=debug EXTRA=hand
kubectl -n tag14 label deploy d4-drift notfall=ja
helm upgrade d4 ./drift -n tag14 --wait
kubectl -n tag14 get deploy d4-drift -o jsonpath="$F"

Expected (abridged):

STATUS: deployed
deployment.apps/d4-drift scaled
deployment.apps/d4-drift env updated
deployment.apps/d4-drift labeled
Error: UPGRADE FAILED: conflict occurred while applying object tag14/d4-drift apps/v1, K
ind=Deployment: Apply failed with 2 conflicts: conflicts with "kubectl" with subresource
 "scale" using apps/v1:
- .spec.replicas
conflicts with "kubectl-set" using apps/v1:
- .spec.template.spec.containers[name="httpbin"].env[name="LOG_LEVEL"].value
4 ja LOG_LEVEL=debug EXTRA=hand

Before the error, Helm 4 prints the same message once more as level=WARN msg="upgrade failed". Instead of silently overwriting, SSA refuses the upgrade and names the field and the field manager. Who owns what is in managedFields:

# on prod
kubectl -n tag14 get deploy d4-drift --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[]
    | [.manager, .operation, .subresource // "-"] | @tsv'
helm upgrade d4 ./drift -n tag14 --force-conflicts --wait | grep REVISION
kubectl -n tag14 get deploy d4-drift -o jsonpath="$F"
helm history d4 -n tag14 -o json | jq -r '.[] | "\(.revision) \(.status)"'

Expected:

helm    Apply   -
kubectl Update  scale
kubectl-set     Update  -
kubectl-label   Update  -
k3s     Update  status
REVISION: 3
2 ja LOG_LEVEL=info EXTRA=hand
1 superseded
2 failed
3 deployed

Apply is SSA, Update a classic write. kubectl scale writes through the scale subresource, kubectl set env is called kubectl-set. --force-conflicts takes the contested fields back; EXTRA and the label still belong to their managers and stay. The result matches Helm 3 – but only after someone has decided on purpose. Day 19 shows the same conflict in operations.

Save your progress? Sign up for free

Drill 4: A Helm 3 release under Helm 4

# on prod
helm get metadata d3 -n tag14 | grep APPLY_METHOD
helm get metadata d4 -n tag14 | grep APPLY_METHOD
kubectl -n tag14 scale deploy d3-drift --replicas=3
helm upgrade d3 ./drift -n tag14 --wait | grep REVISION
helm get metadata d3 -n tag14 | grep APPLY_METHOD
kubectl -n tag14 get deploy d3-drift -o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n tag14 get deploy d3-drift --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[] | select(.manager=="helm") | .operation'

Expected:

APPLY_METHOD: client-side apply (defaulted)
APPLY_METHOD: server-side apply
deployment.apps/d3-drift scaled
REVISION: 3
APPLY_METHOD: client-side apply
2
Update

Helm 3 stores no method – Helm 4 then assumes client-side (“defaulted”). With the default --server-side=auto, Helm 4 sticks with that and behaves like Helm 3: the drift is overwritten without a conflict. Switching is one flag:

# on prod
helm upgrade d3 ./drift -n tag14 --server-side=true --wait | grep REVISION
helm get metadata d3 -n tag14 | grep APPLY_METHOD
kubectl -n tag14 get deploy d3-drift --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[] | select(.manager=="helm") | .operation'

Expected:

REVISION: 4
APPLY_METHOD: server-side apply
Apply

Helm has converted its old Update entry into an Apply entry; from now on auto means server-side. Be careful during the transition if someone still uses Helm 3:

# on prod
helm3 upgrade d4 ./drift -n tag14 --wait | grep REVISION
helm get metadata d4 -n tag14 | grep APPLY_METHOD
helm3 rollback d4 3 -n tag14 | tail -1
helm get metadata d4 -n tag14 | grep APPLY_METHOD
helm upgrade d4 ./drift -n tag14 --server-side=true --wait | grep REVISION

Expected:

REVISION: 4
APPLY_METHOD: client-side apply (defaulted)
Rollback was a success! Happy Helming!
APPLY_METHOD: client-side apply (defaulted)
REVISION: 6

A single helm3 upgrade silently switches an SSA release back to client-side – and so does a Helm 3 rollback to an SSA revision. So: first move all pipelines and admin machines to Helm 4, then switch.

Save your progress? Sign up for free

Drill 5: Field managers and the autoscaler

The usual chart pattern: if an HPA is enabled, the chart does not render replicas. Version 0.2.0 of drift:

# on prod
sed -i 's/^version: 0.1.0/version: 0.2.0/' drift/Chart.yaml
cat > drift/values.yaml <<'EOF'
# drift/values.yaml – Stand Tag 14
replicaCount: 2
logLevel: info
autoscaling:
  enabled: false
  minReplicas: 3
  maxReplicas: 5
EOF
cat > drift/templates/deployment.yaml <<'EOF'
{{/* drift/templates/deployment.yaml – Stand Tag 14 */}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-drift
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
      containers:
        - name: httpbin
          image: ghcr.io/mccutchen/go-httpbin:{{ .Chart.AppVersion }}
          env:
            - name: LOG_LEVEL
              value: {{ .Values.logLevel | quote }}
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
EOF
cat > drift/templates/hpa.yaml <<'EOF'
{{/* drift/templates/hpa.yaml – Stand Tag 14 */}}
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Release.Name }}-drift
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Release.Name }}-drift
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
{{- end }}
EOF
helm template d3 ./drift -s templates/deployment.yaml --set autoscaling.enabled=true \
  | sed -n 3,8p

Expected:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: d3-drift
spec:
  selector:

Now switch a running release to the HPA:

# on prod
helm install h1 ./drift -n tag14 --wait | grep STATUS
helm upgrade h1 ./drift -n tag14 --set autoscaling.enabled=true | grep REVISION
sleep 20
kubectl -n tag14 get events -o custom-columns=MELDUNG:.message \
  --field-selector involvedObject.kind=Deployment,involvedObject.name=h1-drift

Expected:

STATUS: deployed
REVISION: 2
MELDUNG
Scaled up replica set h1-drift-654cfb8d45 from 0 to 2
Scaled down replica set h1-drift-654cfb8d45 from 2 to 1
Scaled up replica set h1-drift-654cfb8d45 from 1 to 3

The column header MELDUNG means “message”. The trap: replicas belonged to Helm alone. When Helm stops rendering the field, SSA removes it – Kubernetes sets the default 1, and only then does the HPA scale back up. Under load, pods are missing at that moment. (The lab has no metrics-server; the HPA can still raise the count to minReplicas.) It works cleanly if the field changes owner first:

# on prod
helm install h2 ./drift -n tag14 --wait | grep STATUS
kubectl -n tag14 scale deploy h2-drift --replicas=3
helm upgrade h2 ./drift -n tag14 --set autoscaling.enabled=true | grep REVISION
sleep 20
kubectl -n tag14 get events -o custom-columns=MELDUNG:.message \
  --field-selector involvedObject.kind=Deployment,involvedObject.name=h2-drift
kubectl -n tag14 get hpa h2-drift

Expected (abridged):

STATUS: deployed
deployment.apps/h2-drift scaled
REVISION: 2
MELDUNG
Scaled up replica set h2-drift-7bd5c8d666 from 0 to 2
Scaled up replica set h2-drift-7bd5c8d666 from 2 to 3
NAME       REFERENCE             TARGETS              MINPODS   MAXPODS   REPLICAS   ...
h2-drift   Deployment/h2-drift   cpu: <unknown>/80%   3         5         3          ...

kubectl scale takes over spec.replicas; during the upgrade Helm only gives up what it no longer owns – nothing drops to 1. The same principle applies to any other actor: if something manages a field permanently (HPA, VPA, an operator), the chart must not render it.

Save your progress? Sign up for free

Drill 6: What else changes

# on prod
helm upgrade d4 ./drift -n tag14 --atomic --force --dry-run=client 2>&1 | head -2
helm install weg ./drift -n tag14 --set replicaCount=0 | grep STATUS
helm uninstall weg -n tag14 --keep-history
helm list -n tag14 -q | tr '\n' ' '; echo
helm3 list -n tag14 -q | tr '\n' ' '; echo
helm3 list -n tag14 -q -a | tr '\n' ' '; echo
mkdir -p dur/templates && printf 'apiVersion: v2\nname: dur\nversion: 0.1.0\n' \
  > dur/Chart.yaml
cat > dur/templates/cm.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: dur
data:
  sekunden: {{ durationSeconds "2m" | quote }}
EOF
helm template dur ./dur | tail -1
helm3 template dur ./dur 2>&1 | head -1

Expected:

Flag --atomic has been deprecated, use --rollback-on-failure instead
Flag --force has been deprecated, use --force-replace instead
STATUS: deployed
release "weg" uninstalled
d3 d4 h1 h2 weg
d3 d4 h1 h2
d3 d4 h1 h2 weg
  sekunden: "120"
Error: parse error at (dur/templates/cm.yaml:6): function "durationSeconds" not defined

The release weg (“gone”) is uninstalled but keeps its history; the key sekunden means “seconds”.

Helm 3 Helm 4 Day
--atomic, --force --rollback-on-failure, --force-replace 13, 19
--wait = everything ready --wait=watcher (kstatus), default hookOnly 13
list only deployed/failed, -a for all list shows every status 1
post-renderer = program post-renderer = plugin 14
client-side patch SSA for new releases 14

Charts that use the duration functions (new in 4.3, Day 7) no longer render under Helm 3. A chart that has to serve both versions does without them, or documents the minimum version with an annotation in Chart.yaml.

Save your progress? Sign up for free

Drill 7: A post-renderer as a plugin

A post-renderer receives the rendered YAML on stdin and prints it modified – here Kustomize (built into kubectl) puts a label team on everything:

# on prod
cd ~/helm/tag14 && mkdir -p team-label
cat > team-label/render.sh <<'EOF'
#!/bin/sh
# team-label/render.sh – Stand Tag 14
set -e
dir=$(mktemp -d)
cat > "$dir/all.yaml"
cat > "$dir/kustomization.yaml" <<'KEOF'
resources: [all.yaml]
labels:
  - pairs: {team: plattform}
    includeSelectors: false
KEOF
kubectl kustomize "$dir"
rm -rf "$dir"
EOF
chmod +x team-label/render.sh
helm3 template d4 ./drift --post-renderer ./team-label/render.sh \
  | grep -A1 'labels:' | head -2
helm template d4 ./drift --post-renderer ./team-label/render.sh 2>&1 | fold -w 88

Expected:

  labels:
    team: plattform
Error: invalid argument "./team-label/render.sh" for "--post-renderer" flag: plugin: {Na
me:./team-label/render.sh Type:postrenderer/v1} not found

Helm 3 takes a path, Helm 4 only the name of a plugin of type postrenderer/v1. The plugin is a plugin.yaml next to the program:

# on prod
cat > team-label/plugin.yaml <<'EOF'
# team-label/plugin.yaml – Stand Tag 14
apiVersion: v1
type: postrenderer/v1
name: team-label
version: 0.1.0
runtime: subprocess
runtimeConfig:
  platformCommand:
    - command: ${HELM_PLUGIN_DIR}/render.sh
EOF
helm plugin install ./team-label
helm plugin list
helm upgrade d4 ./drift -n tag14 --post-renderer team-label --wait | grep REVISION
kubectl -n tag14 get deploy -l team=plattform
helm upgrade d4 ./drift -n tag14 --wait | grep REVISION
kubectl -n tag14 get deploy -l team=plattform
helm3 list -n tag14 -q 2>&1 | head -1 | fold -w 88

Expected:

Installing plugin from local directory (development mode)
Installed plugin: team-label
NAME        VERSION  TYPE             APIVERSION  PROVENANCE  SOURCE
team-label  0.1.0    postrenderer/v1  v1          local dev   unknown
REVISION: 7
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
d4-drift   2/2     2            2           4m18s
REVISION: 8
No resources found in tag14 namespace.
failed to load plugins: failed to load plugin at "/home/ubuntu/.local/share/helm/plugins
/team-label/plugin.yaml": error unmarshaling JSON: while decoding JSON: json: unknown fi
eld "apiVersion"

Three lessons: a local directory counts as “local dev” and needs no signature. Helm does not store the post-renderer in the release – the next upgrade without the flag removes the label again. And because both versions use the same plugin directory, Helm 3 complains about every plugin in the new format (apiVersion: v1). Besides subprocesses, Helm 4 also knows WebAssembly plugins (runtime: extism/v1) – not needed in the lab.

Save your progress? Sign up for free

Drill 8: Install helm-diff with signature check

By default, Helm 4 verifies plugin tarballs against a .prov file (Day 16 shows the same for charts). helm-diff ships both:

# on prod
mkdir -p ~/helm/tag14/diff && cd ~/helm/tag14/diff
ARCH=$(dpkg --print-architecture)
U=https://github.com/databus23/helm-diff/releases/download/v3.15.15
for f in diff-3.15.15-linux-$ARCH.tgz diff-3.15.15-linux-$ARCH.tgz.prov \
  helm-diff_3.15.15_checksums.txt pubkey.asc; do curl -fsSLO $U/$f; done
grep " diff-3.15.15-linux-$ARCH.tgz$" helm-diff_3.15.15_checksums.txt | sha256sum -c
helm plugin install diff-3.15.15-linux-$ARCH.tgz 2>&1 | tail -1 | fold -w 88
gpg --show-keys pubkey.asc | sed -n 2p
gpg --no-default-keyring --keyring ./helm-diff.kbx --import pubkey.asc 2>&1 | tail -1
helm plugin verify diff-3.15.15-linux-$ARCH.tgz --keyring ./helm-diff.kbx
helm plugin install diff-3.15.15-linux-$ARCH.tgz --keyring ./helm-diff.kbx | tail -1
helm plugin list | cut -c1-70

Expected (abridged):

diff-3.15.15-linux-amd64.tgz: OK
Error: plugin verification failed: reading keybox /home/ubuntu/.gnupg/pubring.kbx: keybo
x contains no OpenPGP keys
      C5645EF47482257A1F806D2BEA17A2A206AFF8CD
gpg:               imported: 1
Signed by: Fabian Ruff (helm-diff release key) <…>
Using Key With Fingerprint: C5645EF47482257A1F806D2BEA17A2A206AFF8CD
Plugin Hash Verified: sha256:504e9941…
Installed plugin: diff
NAME        VERSION  TYPE             APIVERSION  PROVENANCE             SO
diff        3.15.15  cli/v1           legacy      mismatched provenance  un
team-label  0.1.0    postrenderer/v1  v1          local dev              un

Without a key the installation aborts – good. With a dedicated keyring that contains only the release key, it works. Compare the fingerprint with the project page. “mismatched provenance” in the list is cosmetic: the .prov names diff-3.15.15-linux-amd64.tgz, Helm stores the copy as diff-3.15.15.tgz. legacy means: old plugin format, runs under both versions. Now the payoff:

# on prod
cd ~/helm/tag14
helm diff upgrade d4 ./drift -n tag14 --set logLevel=debug -C 1
kubectl -n tag14 set env deploy/d4-drift LOG_LEVEL=warn
helm diff upgrade d4 ./drift -n tag14 -C 1 --detailed-exitcode; echo "Exit: $?"
helm diff upgrade d4 ./drift -n tag14 -C 1 --three-way-merge --detailed-exitcode
echo "Exit: $?"
helm upgrade d4 ./drift -n tag14 2>&1 | grep '^Error' \
  | grep -o 'conflict with "[a-z-]*"'
helm upgrade d4 ./drift -n tag14 --force-conflicts --wait | grep STATUS

Expected (abridged):

tag14, d4-drift, Deployment (apps) has changed:
...
              - name: LOG_LEVEL
-               value: "info"
+               value: "debug"
...
deployment.apps/d4-drift env updated
Exit: 0
tag14, d4-drift, Deployment (apps) has changed:
...
          - name: LOG_LEVEL
-           value: warn
+           value: info
...
Error: identified at least one change, exiting with non-zero exit code (…)
Error: plugin "diff" exited with error
Exit: 2
conflict with "kubectl-set"
STATUS: deployed

Without --three-way-merge, helm-diff compares the stored manifest with the new one – the drift stays invisible (exit 0). With the flag it compares against the live object and shows exactly the field the SSA upgrade then fails on. --detailed-exitcode (2 = changes) turns this into a pipeline stage.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the apply method for all releases in tag14. Target time: 30 s
  2. Who owns spec.replicas of h2-drift? Target time: 30 s
  3. List the installed plugins with their type. Target time: 15 s
  4. Does drift with HPA render the same under Helm 3 and Helm 4? Ignore blank lines. Target time: 30 s

Solution

# on prod
for r in d3 d4 h1 h2; do echo "$r $(helm get metadata $r -n tag14 | grep APPLY_METHOD)"
done
kubectl -n tag14 get deploy h2-drift --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[]
    | select(.fieldsV1."f:spec"."f:replicas") | .manager'
helm plugin list | awk 'NR>1 {print $1, $3}'
diff -B <(helm template x ./drift --set autoscaling.enabled=true) \
  <(helm3 template x ./drift --set autoscaling.enabled=true 2>/dev/null) \
  && echo identisch

Expected:

d3 APPLY_METHOD: server-side apply
d4 APPLY_METHOD: server-side apply
h1 APPLY_METHOD: server-side apply
h2 APPLY_METHOD: server-side apply
kubectl
diff cli/v1
team-label postrenderer/v1
identisch

identisch means “identical”. Without -B, diff reports one line: Helm 4 puts a blank line before every ---. If you compare renderings from different Helm versions in CI, this will trip you up.

Save your progress? Sign up for free

Checkpoint

# on prod
helm3 version --short 2>/dev/null

Expected: v3.22.1+gd6c1a0e

# on prod
helm get metadata d3 -n tag14 | grep APPLY_METHOD

Expected: APPLY_METHOD: server-side apply – the Helm 3 release has been switched.

# on prod
helm diff version

Expected: 3.15.15

# on prod
helm status notes -n notes -o json | jq -r .info.status

Expected: deployed – notes was not touched today.

Challenge

Install c with Helm 3, scale it to 3 by hand and switch it to Server-Side Apply with Helm 4. Beforehand, it must be visible what changes, and the history must not contain a failed attempt.

Hint: helm-diff with --three-way-merge shows the drift; one upgrade can carry --server-side=true and --force-conflicts at the same time.

Solution

# on prod
helm3 install c ./drift -n tag14 --wait 2>/dev/null | grep STATUS
kubectl -n tag14 scale deploy c-drift --replicas=3
helm diff upgrade c ./drift -n tag14 --three-way-merge -C 0 | grep '^[-+]'
helm upgrade c ./drift -n tag14 --server-side=true --force-conflicts --wait \
  | grep REVISION
helm get metadata c -n tag14 | grep APPLY_METHOD
kubectl -n tag14 get deploy c-drift -o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n tag14 get deploy c-drift --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[] | [.manager, .operation] | @tsv'
helm history c -n tag14 -o json | jq -r '.[] | "\(.revision) \(.status)"'

Expected:

STATUS: deployed
deployment.apps/c-drift scaled
-   replicas: 3
+   replicas: 2
REVISION: 2
APPLY_METHOD: server-side apply
2
helm    Apply
k3s     Update
1 superseded
2 deployed

2>/dev/null hides the Helm 3 plugin warning from Drill 7. Without --force-conflicts, the first SSA upgrade would have failed on the manager kubectl – and would still have switched the method to server-side. The diff beforehand is the conscious decision: the night’s 3 replicas disappear.

Save your progress? Sign up for free

Cleanup

# on prod
helm uninstall d3 d4 h1 h2 c weg -n tag14
helm plugin uninstall team-label
kubectl delete namespace tag14

Expected:

release "d3" uninstalled
release "d4" uninstalled
release "h1" uninstalled
release "h2" uninstalled
release "c" uninstalled
release "weg" uninstalled
Uninstalled plugin: team-label
namespace "tag14" deleted

helm3 and helm-diff stay installed. Nothing has changed in the chart notes; there is no commit.

Interview Note

  • What happens to a manual change on the next helm upgrade? Helm 3: fields the chart renders are silently reset (three-way merge), foreign fields stay. Helm 4 with SSA: a conflict naming the field and the field manager; only --force-conflicts takes over.
  • How do you migrate from Helm 3 to Helm 4? First move all clients to Helm 4 (Helm 3 switches SSA releases back), check plugins for the new format, make drift visible with helm diff --three-way-merge, then run --server-side=true once per release.
  • Why does a chart with an HPA not render replicas? Otherwise Helm and the HPA fight over the field. When switching, the field has to change owner, or it briefly drops to 1.

Enterprise Note

  • Pin the Helm version centrally in pipelines, admin images and GitOps tools; one forgotten Helm 3 runner silently undoes the switch.
  • SSA conflicts are a signal, not a disruption: instead of adding --force-conflicts to every pipeline across the board, build in drift checks (helm-diff, GitOps drift detection) and clarify who should really manage the field.
  • Install plugins only with a verified signature, keep the keys in a separate keyring per plugin, document the fingerprints. --verify=false does not belong in automation.
  • Post-renderers are invisible to helm get – the result belongs in the review (helm get manifest after the deployment), the plugin in every pipeline.
  • Prefer installing charts by digest rather than by version once they live in a registry (Day 15).