Skip to content
Hands-on IT IT Drills

Day 19: Troubleshooting

Goal: Assign every Helm error to a layer, resolve stuck and failed releases cleanly, and fix the typical API errors without losing data. · Duration: approx. 90 min · Prerequisite: Day 18

Briefing

  • Three layers: rendering (errors with file and line, no cluster needed), API (the server rejects), runtime (everything accepted, but the Pods do not run).
  • The first minute changes nothing: status, history, get values, get manifest, events.
  • Helm locks a release through the revision with status pending-*. If Helm is killed hard, the lock stays.
  • Many Helm errors are Kubernetes errors: immutable fields, foreign objects, field manager conflicts, removed APIs.
  • Back up first, then repair – release Secrets above all.

Drill 1: Three layers

All experiments run in tag19, with helper and small throwaway charts.

# on prod
mkdir -p ~/helm/tag19 && cd ~/helm/charts
kubectl create namespace tag19
helm template t ./notes 2>&1 | head -1
helm install a ./helper -n tag19 --set port=abc 2>&1 | cut -c1-80
helm list -n tag19 --failed -q
kubectl -n tag19 get svc,deploy -l app.kubernetes.io/instance=a -o name
helm template a ./helper --set port=abc | kubectl apply --dry-run=server -f - 2>&1 \
  | cut -c1-80
helm install l ./helper -n tag19 --set image.tag=2.99.0 | grep STATUS
sleep 10
kubectl -n tag19 get pods -l app.kubernetes.io/instance=l

Expected (abridged):

namespace/tag19 created
Error: execution error at (notes/templates/deployment.yaml:35:27): apiKey.existingSe…
Error: INSTALLATION FAILED: server-side apply failed for object tag19/a-helper a
a
service/a-helper
service/a-helper created (server dry run)
Error from server (BadRequest): error when creating "STDIN": Deployment in versi
STATUS: deployed
NAME                       READY   STATUS         RESTARTS   AGE
l-helper-8bcc4b48c-r4qz4   0/1     ErrImagePull   0          10s

The first error is the required check of notes (apiKey.existingSecret fehlt, ‘missing’), which fires because no values file is given.

Layer Sign Tool
Rendering execution error at (file:line) helm template --debug
API apply failed, invalid, conflict kubectl apply --dry-run=server
Runtime deployed, but Pods not ready kubectl get/describe, events, logs

The failed install a leaves behind a failed release and a Service – Helm does not clean up half-created objects. helm install --dry-run=server would not have found the error (Day 4), kubectl apply --dry-run=server does.

Save your progress? Sign up for free

Drill 2: The first minute

Release l reports deployed and still does not run. Five sources, no changes:

# on prod
helm status l -n tag19 | grep -E '^(STATUS|REVISION|DESCRIPTION)'
helm status l -n tag19 | grep -A2 'v1/Pod'
helm history l -n tag19
helm get values l -n tag19
helm get values l -n tag19 --all | head -4
helm get manifest l -n tag19 | grep 'image:'
kubectl -n tag19 get events --field-selector reason=Failed \
  -o jsonpath='{.items[0].message}' | fold -w 88; echo

Expected (abridged):

STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
==> v1/Pod(related)
NAME                       READY   STATUS         RESTARTS   AGE
l-helper-8bcc4b48c-r4qz4   0/1     ErrImagePull   0          11s
REVISION  UPDATED                   STATUS    CHART         ...  DESCRIPTION
1         Fri Nov 20 07:25:52 2026  deployed  helper-0.2.1  ...  Install complete
USER-SUPPLIED VALUES:
image:
  tag: 2.99.0
COMPUTED VALUES:
image:
  repository: ghcr.io/mccutchen/go-httpbin
  tag: 2.99.0
          image: "ghcr.io/mccutchen/go-httpbin:2.99.0"
Failed to pull image "ghcr.io/mccutchen/go-httpbin:2.99.0": rpc error: code = NotFound d
esc = failed to pull and unpack image "ghcr.io/mccutchen/go-httpbin:2.99.0": failed to r
esolve reference "ghcr.io/mccutchen/go-httpbin:2.99.0": ghcr.io/mccutchen/go-httpbin:2.9
9.0: not found

In Helm 4, status also shows the Pods. history names the last good revision and often already the reason in DESCRIPTION. get values without --all shows only what someone set – here a tag that does not exist. This is a runtime error: Helm did everything right.

Save your progress? Sign up for free

Drill 3: Stuck upgrade – pending-upgrade

Repair first, then start an upgrade with --wait and kill Helm hard – like a CI runner that ends with SIGKILL:

# on prod
helm upgrade l ./helper -n tag19 --set image.tag=2.26.0 --wait | grep REVISION
helm upgrade l ./helper -n tag19 --set image.tag=2.98.0 --wait --timeout 5m \
  >/dev/null 2>&1 &
sleep 4; kill -9 $!
helm history l -n tag19 --max 2
helm upgrade l ./helper -n tag19 --set image.tag=2.26.0 2>&1 | tail -1
pgrep -a helm || echo "kein helm-Prozess"
kubectl -n tag19 get deploy l-helper \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

kein helm-Prozess means ‘no helm process’.

Expected (abridged):

REVISION: 2
REVISION  UPDATED                   STATUS           ...  DESCRIPTION
2         Fri Nov 20 07:26:19 2026  deployed         ...  Upgrade complete
3         Fri Nov 20 07:26:23 2026  pending-upgrade  ...  Preparing upgrade
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
kein helm-Prozess
ghcr.io/mccutchen/go-httpbin:2.98.0

Revision 3 is the orphaned lock. The cluster has already taken the broken image. Same order as in operations: is there really no Helm running any more (not in another pipeline either)? How is the application doing? Then the clean way – roll back to the last deployed revision:

# on prod
helm rollback l 2 -n tag19 --wait --description "pending-upgrade aufgelöst"
helm history l -n tag19 --max 3
helm upgrade l ./helper -n tag19 --set image.tag=2.26.0 --wait | grep REVISION

The description means ‘pending-upgrade resolved’.

Expected (abridged):

Rollback was a success! Happy Helming!
REVISION  ...  STATUS           ...  DESCRIPTION
2         ...  superseded       ...  Upgrade complete
3         ...  pending-upgrade  ...  Preparing upgrade
4         ...  deployed         ...  pending-upgrade aufgelöst
REVISION: 5

Helm 4 allows the rollback despite the lock; revision 3 stays in the history as pending-upgrade but no longer gets in the way. The last resort, if the rollback does not work: delete exactly the Secret of the stuck revision.

Warning: Delete only the newest pending-* Secret, and back it up first. The label filter also finds the old, harmless revision 3 – so sort and double-check the name. If you delete all release Secrets, you delete Helm’s memory: the release is unknown afterwards, and the objects keep running without an owner.

# on prod
helm upgrade l ./helper -n tag19 --set image.tag=2.98.0 --wait --timeout 5m \
  >/dev/null 2>&1 &
sleep 4; kill -9 $!
kubectl -n tag19 get secret -l owner=helm,name=l,status=pending-upgrade \
  --sort-by=.metadata.creationTimestamp -o name
kubectl -n tag19 get secret sh.helm.release.v1.l.v6 -o yaml > ~/helm/tag19/l-v6.yaml
kubectl -n tag19 delete secret sh.helm.release.v1.l.v6
helm history l -n tag19 --max 1
kubectl -n tag19 get deploy l-helper \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
helm upgrade l ./helper -n tag19 --set image.tag=2.26.0 --wait | grep REVISION

Expected (abridged):

secret/sh.helm.release.v1.l.v3
secret/sh.helm.release.v1.l.v6
secret "sh.helm.release.v1.l.v6" deleted from tag19 namespace
REVISION  ...  STATUS    ...  DESCRIPTION
5         ...  deployed  ...  Upgrade complete
ghcr.io/mccutchen/go-httpbin:2.98.0
REVISION: 6

Deleting the Secret changes nothing in the cluster – the Deployment still runs 2.98.0; only the next upgrade repairs it. Prevention: only one pipeline per release at a time, and a --timeout shorter than the CI job timeout.

Save your progress? Sign up for free

Drill 4: Failed first install

The same hard abort, this time during the install:

# on prod
helm install pi ./helper -n tag19 --set image.tag=2.98.0 --wait --timeout 5m \
  >/dev/null 2>&1 &
sleep 4; kill -9 $!
helm history pi -n tag19
helm upgrade --install pi ./helper -n tag19 2>&1 | tail -1
helm rollback pi -n tag19 2>&1 | tail -1
helm uninstall pi -n tag19

Expected (abridged):

REVISION  UPDATED                   STATUS           ...  DESCRIPTION
1         Fri Nov 20 07:26:39 2026  pending-install  ...  Initial install underway
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Error: release has no 0 version
release "pi" uninstalled

There is no good revision a rollback could lead to – uninstall and install again (check first whether data is affected). Release a from Drill 1 is different; its only revision is failed:

# on prod
helm history a -n tag19 | cut -c1-88
helm upgrade a ./helper -n tag19 2>&1 | grep -o 'containerPort="abc"' | head -1
helm get values a -n tag19
helm upgrade a ./helper -n tag19 --reset-values --wait | grep -E '^(STATUS|REVISION)'
helm history a -n tag19 -o json | jq -r '.[] | "\(.revision) \(.status)"'

Expected:

REVISION  UPDATED                   STATUS  CHART         ...  DESCRIPTION
1         Fri Nov 20 07:25:50 2026  failed  helper-0.2.1  ...  Release "a" failed: se
containerPort="abc"
USER-SUPPLIED VALUES:
port: abc
STATUS: deployed
REVISION: 3
1 failed
2 superseded
3 deployed

The message has no deployed releases from Helm 3 days does not appear here in Helm 4.3.2: an upgrade of a release that has only failed revisions is allowed. The trap is elsewhere: an upgrade without values takes over the values of the last deployed revision (Day 3) – if there is none, those of the failed one (Day 20, round 4). The broken port comes back with it; only --reset-values helps.

Save your progress? Sign up for free

Drill 5: field is immutable

A StatefulSet with data, and then the PVC is supposed to grow:

# on prod
kubectl -n tag19 create secret generic pg-db --from-literal=password=lab-db-123
helm install pg ./postgres -n tag19 --set auth.existingSecret=pg-db --wait | grep STATUS
kubectl -n tag19 exec pg-postgres-0 -- psql -U notes -qc \
  'create table t (x int); insert into t values (19)'
helm upgrade pg ./postgres -n tag19 --reuse-values --set storage=2Gi 2>&1 \
  | grep -o 'spec.volumeClaimTemplates: Invalid value\|field is immutable' | sort -u

Expected:

secret/pg-db created
STATUS: deployed
field is immutable
spec.volumeClaimTemplates: Invalid value

volumeClaimTemplates, selector and serviceName of a StatefulSet are immutable. The way out: delete the StatefulSet, but leave the Pods and PVCs in place.

Warning: kubectl delete statefulset --cascade=orphan keeps Pods and PVCs running. Without --cascade=orphan, Kubernetes deletes the Pods – the database is gone until the upgrade starts it again.

# on prod
kubectl -n tag19 delete statefulset pg-postgres --cascade=orphan
kubectl -n tag19 get pods -l app.kubernetes.io/instance=pg
helm upgrade pg ./postgres -n tag19 --reuse-values --set storage=2Gi --wait \
  | grep REVISION
kubectl -n tag19 get pods -l app.kubernetes.io/instance=pg
kubectl -n tag19 get sts pg-postgres \
  -o jsonpath='{.spec.volumeClaimTemplates[0].spec.resources.requests.storage}{"\n"}'
kubectl -n tag19 get pvc data-pg-postgres-0 \
  -o jsonpath='{.status.capacity.storage}{"\n"}'
kubectl -n tag19 exec pg-postgres-0 -- psql -U notes -tAc 'select x from t'

Expected:

statefulset.apps "pg-postgres" deleted from tag19 namespace
NAME            READY   STATUS    RESTARTS   AGE
pg-postgres-0   1/1     Running   0          22s
REVISION: 3
NAME            READY   STATUS    RESTARTS   AGE
pg-postgres-0   1/1     Running   0          24s
2Gi
1Gi
19

The new StatefulSet adopts the running Pod (same age, no restart), and the data stays. But: the existing PVC stays at 1Gi – volumeClaimTemplates applies only to new PVCs. You can grow it only on the PVC itself, and only if the StorageClass supports allowVolumeExpansion (local-path does not).

Second classic: a Job as a normal object whose command changes.

Warning: --force-replace is not a normal upgrade flag. It overwrites objects completely (including fields set by controllers) and switches the release to client-side apply (Day 14).

In the throwaway chart, sekunden means ‘seconds’, einmal ‘once’ and schlafe ‘sleep’:

# on prod
cd ~/helm/tag19 && mkdir -p jobchart/templates
printf 'apiVersion: v2\nname: jobchart\nversion: 0.1.0\n' > jobchart/Chart.yaml
printf 'sekunden: "1"\n' > jobchart/values.yaml
cat > jobchart/templates/job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-einmal
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: schlafe
          image: curlimages/curl:8.23.0
          command: ["sleep", {{ .Values.sekunden | quote }}]
EOF
helm install j ./jobchart -n tag19 --wait --wait-for-jobs | grep STATUS
helm upgrade j ./jobchart -n tag19 --set-string sekunden=2 2>&1 | tail -1 | cut -c1-88
helm upgrade j ./jobchart -n tag19 --set-string sekunden=2 --force-replace 2>&1 \
  | tail -1
helm upgrade j ./jobchart -n tag19 --set-string sekunden=2 --force-replace \
  --server-side=false 2>&1 | tail -1 | cut -c1-88
kubectl -n tag19 delete job j-einmal
helm upgrade j ./jobchart -n tag19 --set-string sekunden=2 --wait --wait-for-jobs \
  | grep REVISION
helm get metadata j -n tag19 | grep APPLY_METHOD
helm upgrade j ./jobchart -n tag19 --server-side=true --reuse-values | grep REVISION
helm get metadata j -n tag19 | grep APPLY_METHOD

Expected:

STATUS: deployed
Error: UPGRADE FAILED: server-side apply failed for object tag19/j-einmal batch/v1, Kind
Error: UPGRADE FAILED: invalid operation: cannot use server-side apply and force replace
together
Error: UPGRADE FAILED: failed to replace object: Job.batch "j-einmal" is invalid: [spec.
job.batch "j-einmal" deleted from tag19 namespace
REVISION: 5
APPLY_METHOD: client-side apply
REVISION: 6
APPLY_METHOD: server-side apply

Three findings. --force-replace (formerly --force) does not work together with Server-Side Apply. With --server-side=false, Helm replaces the object with a PUT – it does not delete and recreate it, so immutable fields stay immutable. And: the failed attempt with --server-side=false switched the release to client-side, because auto takes over the method of the previous revision; only --server-side=true switches back. For Jobs, the clean solution is a hook with before-hook-creation (Day 11) or a name that contains the revision.

Save your progress? Sign up for free

Drill 6: Who owns the object?

A chart with fixed names – like many charts from the internet. The names are German: besitz ‘ownership’, gemeinsam ‘shared’, von ‘from’, hand ‘by hand’, notes-leser ‘notes reader’:

# on prod
cd ~/helm/tag19 && mkdir -p besitz/templates
printf 'apiVersion: v2\nname: besitz\nversion: 0.1.0\n' > besitz/Chart.yaml
cat > besitz/templates/objekte.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: gemeinsam
data:
  von: {{ .Release.Name }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: notes-leser
rules:
  - apiGroups: [""]
    resources: [configmaps]
    verbs: [get, list]
EOF
kubectl -n tag19 create configmap gemeinsam --from-literal=von=hand
helm install b1 ./besitz -n tag19 2>&1 | fold -w 88
helm list -n tag19 -q | grep -c b1

Expected:

configmap/gemeinsam created
Error: INSTALLATION FAILED: unable to continue with install: ConfigMap "gemeinsam" in na
mespace "tag19" exists and cannot be imported into the current release: invalid ownershi
p metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be
set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": mus
t be set to "b1"; annotation validation error: missing key "meta.helm.sh/release-namespa
ce": must be set to "tag19"
0

Helm adopts only objects with the label app.kubernetes.io/managed-by: Helm and the annotations meta.helm.sh/release-name and -namespace. The check runs beforehand – not even a release is created. --take-ownership skips it:

# on prod
helm install b1 ./besitz -n tag19 --take-ownership 2>&1 | tail -1 | fold -w 88
kubectl -n tag19 get cm gemeinsam --show-managed-fields -o json \
  | jq -c '.metadata.managedFields[] | {manager, operation, fields: .fieldsV1}'
helm upgrade b1 ./besitz -n tag19 --take-ownership --force-conflicts | grep REVISION
kubectl -n tag19 get cm gemeinsam -o jsonpath='{.data.von}{"\n"}'
kubectl -n tag19 get cm gemeinsam --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[].manager'

Expected:

Error: INSTALLATION FAILED: conflict occurred while applying object tag19/gemeinsam /v1,
 Kind=ConfigMap: Apply failed with 1 conflict: conflict with "kubectl-create" using v1:
.data.von
{"manager":"kubectl-create","operation":"Update","fields":{"f:data":{".":{},"f:von":…
REVISION: 2
b1
helm
kubectl-create

The ownership check is gone, and now Server-Side Apply speaks up (Day 14): the field data.von belongs to the field manager kubectl-create, and Helm wants a different value. Who owns what is in managedFields. --force-conflicts hands the field to Helm. With cluster-wide objects, two releases collide:

# on prod
kubectl create namespace tag19b
helm install b2 ./besitz -n tag19b 2>&1 | grep -o 'ClusterRole.*current value is "b1"' \
  | fold -w 88
sed -i 's/name: notes-leser/name: {{ .Release.Name }}-notes-leser/' \
  besitz/templates/objekte.yaml
kubectl -n tag19b create configmap gemeinsam --from-literal=von=b2
kubectl -n tag19b label configmap gemeinsam app.kubernetes.io/managed-by=Helm
kubectl -n tag19b annotate configmap gemeinsam meta.helm.sh/release-name=b2 \
  meta.helm.sh/release-namespace=tag19b
helm install b2 ./besitz -n tag19b | grep STATUS
kubectl get clusterrole -l app.kubernetes.io/managed-by=Helm -o name | grep leser

Expected:

namespace/tag19b created
ClusterRole "notes-leser" in namespace "" exists and cannot be imported into the current
 release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/rel
ease-name" must equal "b2": current value is "b1"
configmap/gemeinsam created
configmap/gemeinsam labeled
configmap/gemeinsam annotated
STATUS: deployed
clusterrole.rbac.authorization.k8s.io/b2-notes-leser
clusterrole.rbac.authorization.k8s.io/notes-leser

The right repair is in the chart: cluster-wide objects get the release name. Helm adopts the hand-made ConfigMap in tag19b after label and annotate without a conflict – it already has the value the chart sets.

Save your progress? Sign up for free

Drill 7: Conflict after kubectl scale

Someone scales by hand during the night. In the morning the next upgrade comes – without any change:

# on prod
cd ~/helm/charts
kubectl -n tag19 scale deployment l-helper --replicas=3
helm upgrade l ./helper -n tag19 --reuse-values 2>&1 | tail -1 | fold -w 88
kubectl -n tag19 get deploy l-helper --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[]
    | select(.fieldsV1."f:spec"."f:replicas") | [.manager, .subresource] | @tsv'
helm upgrade l ./helper -n tag19 --reuse-values --set replicaCount=2 --force-conflicts \
  --wait | grep REVISION
kubectl -n tag19 get deploy l-helper -o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n tag19 get deploy l-helper --show-managed-fields -o json \
  | jq -r '.metadata.managedFields[]
    | select(.fieldsV1."f:spec"."f:replicas") | [.manager, .operation] | @tsv'

Expected:

deployment.apps/l-helper scaled
Error: UPGRADE FAILED: conflict occurred while applying object tag19/l-helper apps/v1, K
ind=Deployment: Apply failed with 1 conflict: conflict with "kubectl" with subresource "
scale" using apps/v1: .spec.replicas
kubectl scale
REVISION: 8
2
helm    Apply

With Server-Side Apply (the default for new installs in Helm 4, Day 14), spec.replicas belongs to the manager kubectl after kubectl scale. The chart always renders replicas, so every upgrade fails. Either Helm takes the field back with --force-conflicts, or the chart leaves out replicas when an HPA or a human scales – with an ownership handover when you switch, otherwise the Deployment briefly drops to one Pod (Day 14, Drill 5). If you do not know this, all you see is a pipeline that suddenly turns red. helm diff upgrade --three-way-merge (Day 14) would have shown the drift beforehand.

Save your progress? Sign up for free

Drill 8: Removed API in the stored release

After a Kubernetes upgrade, the stored manifest can contain an API that the cluster no longer knows. On k3s 1.37 you can only reproduce this by manipulating the release Secret (policy/v1beta1 has been gone since 1.25). kaputt means ‘broken’:

# on prod
cd ~/helm/tag19 && mkdir -p altapi/templates
printf 'apiVersion: v2\nname: altapi\nversion: 0.1.0\n' > altapi/Chart.yaml
cat > altapi/templates/pdb.yaml <<'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ .Release.Name }}-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ .Release.Name }}
EOF
helm install alt ./altapi -n tag19 | grep STATUS
S=sh.helm.release.v1.alt.v1
kubectl -n tag19 get secret $S -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gzip -d \
  | jq '.manifest |= sub("policy/v1"; "policy/v1beta1")' \
  | gzip -c | base64 -w0 | base64 -w0 > kaputt.b64
printf '{"data":{"release":"%s"}}' "$(cat kaputt.b64)" > kaputt.json
kubectl -n tag19 patch secret $S --type merge --patch-file kaputt.json
helm get manifest alt -n tag19 | grep apiVersion
helm upgrade alt ./altapi -n tag19 2>&1 | fold -w 88
helm history alt -n tag19 -o json | jq -r '.[] | "\(.revision) \(.status)"'

Expected:

STATUS: deployed
secret/sh.helm.release.v1.alt.v1 patched
apiVersion: policy/v1beta1
Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest:
 resource mapping not found for name: "alt-pdb" namespace: "" from "": no matches for ki
nd "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first
1 deployed

The chart is fine; Helm already fails while reading the old state – and does not even create a failed revision. The plugin helm-mapkubeapis rewrites such manifests automatically; it is not installed in the lab, and whether version 0.6.1 runs reliably with Helm 4 is not established. You can find such releases beforehand with pluto detect-helm (Day 17). By hand it is the same route backwards (sicherung means ‘backup’, repariert ‘repaired’):

Warning: Rewriting a release Secret by hand is open-heart surgery. Back it up first (-o yaml), touch only the latest revision, and keep the double Base64 encoding and gzip exactly. A patch file instead of -p avoids overlong command lines for large releases.

# on prod
kubectl -n tag19 get secret $S -o yaml > alt-v1-sicherung.yaml
kubectl -n tag19 get secret $S -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gzip -d \
  | jq '.manifest |= gsub("policy/v1beta1"; "policy/v1")' \
  | gzip -c | base64 -w0 | base64 -w0 > repariert.b64
printf '{"data":{"release":"%s"}}' "$(cat repariert.b64)" > repariert.json
kubectl -n tag19 patch secret $S --type merge --patch-file repariert.json
helm get manifest alt -n tag19 | grep apiVersion
helm upgrade alt ./altapi -n tag19 | grep -E '^(STATUS|REVISION)'

Expected:

secret/sh.helm.release.v1.alt.v1 patched
apiVersion: policy/v1
STATUS: deployed
REVISION: 2

Prevention is cheaper: before every Kubernetes upgrade, render the charts against the new version with helm template --kube-version (Day 4), and bring the releases to current APIs with an upgrade while the old cluster is still running.

Message Cause Drill
another operation … is in progress orphaned pending-* revision 3, 4
release has no 0 version rollback without a good revision 4
field is immutable selector, volumeClaimTemplates, Job template 5
invalid ownership metadata object belongs to no release / another release 6
conflict with "…" using … another field manager owns the field 6, 7
unable to build kubernetes objects … removed API in the stored release 8
context deadline exceeded image, probe, hook Job, PVC Challenge

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all releases in the cluster whose current revision is failed or pending-*. Target time: 15 s
  2. Find all release Secrets in the cluster with status pending-upgrade. Target time: 20 s
  3. Which revision of l failed last, and with which values? Target time: 45 s
  4. Which release owns the ConfigMap gemeinsam in tag19? Target time: 20 s

Solution

# on prod
helm list -A --failed --pending
kubectl get secret -A -l owner=helm,status=pending-upgrade -o name
helm history l -n tag19 -o json | jq -r '[.[] | select(.status=="failed")][-1].revision'
helm get values l -n tag19 --revision 7
kubectl -n tag19 get cm gemeinsam -o jsonpath='{.metadata.annotations}{"\n"}'

Expected:

NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION
secret/sh.helm.release.v1.l.v3
7
USER-SUPPLIED VALUES:
image:
  tag: 2.26.0
{"meta.helm.sh/release-name":"b1","meta.helm.sh/release-namespace":"tag19"}

helm list filters by the status of the current revision – all releases are repaired, so the list is empty. The Secret l.v3, on the other hand, is the old lock from Drill 3: harmless, but an alert on “pending Secrets exist” would fire falsely here. Revision 7 was the upgrade without changes that failed because of kubectl scale.

Save your progress? Sign up for free

Checkpoint

# on prod
helm list -A --failed --pending | wc -l

Expected: 1 (only the header line).

# on prod
helm list -n tag19 -o json | jq -r '.[] | "\(.name) \(.status)"' | sort | tr '\n' ' '

Expected: a deployed alt deployed b1 deployed j deployed l deployed pg deployed (after the Challenge also c deployed).

# on prod
helm status notes -n notes | grep -E '^STATUS'

Expected: STATUS: deployed – the prod release noticed none of this.

Challenge

A colleague sent off this command, and the pipeline reports context deadline exceeded. Find the cause without looking at the command, and bring c to deployed.

# on prod
cd ~/helm/charts
helm upgrade --install c ./helper -n tag19 --set probePath=/status/503 \
  --wait --timeout 30s 2>&1 | tail -2

Expected:

Error: resource Deployment/tag19/c-helper not ready. status: InProgress, message: Ava…
context deadline exceeded

Hint: context deadline exceeded only means “Helm waited” – the answer is in the Pods and their events.

Solution

# on prod
helm status c -n tag19 | grep -E '^STATUS'
kubectl -n tag19 get pods -l app.kubernetes.io/instance=c
kubectl -n tag19 describe pod -l app.kubernetes.io/instance=c | grep -m1 'statuscode'
helm get values c -n tag19
helm upgrade c ./helper -n tag19 --reset-values --wait --timeout 60s | grep REVISION
helm history c -n tag19 -o json | jq -r '.[] | "\(.revision) \(.status)"'

Expected (abridged):

STATUS: failed
NAME                        READY   STATUS    RESTARTS   AGE
c-helper-7bd5bf5496-rbsm2   0/1     Running   0          31s
  Warning  Unhealthy  8s (x5 over 30s)  kubelet  spec.containers{helper}: Readiness
probe failed: HTTP probe failed with statuscode: 503
USER-SUPPLIED VALUES:
probePath: /status/503
REVISION: 2
1 superseded
2 deployed

The Pod runs but is never ready: the readinessProbe asks for /status/503. The cause was in get values, the proof in the events. --reset-values discards the wrong value (an upgrade without values would have taken it over from the failed revision, Drill 4). The other candidates for the same error: image cannot be pulled (ErrImagePull, Drill 2), hook Job stuck (kubectl get jobs, Day 11), PVC Pending (kubectl get pvc, events of the PVC).

Save your progress? Sign up for free

Cleanup

# on prod
helm uninstall -n tag19 $(helm list -n tag19 -q)
helm uninstall b2 -n tag19b
kubectl delete namespace tag19 tag19b
kubectl get clusterrole -o name | grep -c notes-leser

Expected (abridged):

release "a" uninstalled
…
release "pg" uninstalled
release "b2" uninstalled
namespace "tag19" deleted
namespace "tag19b" deleted
0

The PVC data-pg-postgres-0 survives helm uninstall (Day 9) and only disappears with the namespace. Nothing changed in the notes chart today, so there is no commit.

Interview Note

  • A Helm upgrade fails. How do you proceed? Determine the layer (rendering, API, runtime), then status, history, get values, get manifest, events and logs – still without changing anything. Back up first, then roll back or repair.
  • another operation is in progress – what do you do? Check that really no run is active, check the state of the application, roll back to the last deployed revision. Only if that does not work: back up and delete the Secret of the newest pending revision. Afterwards fix the cause in CI (lock, timeouts, no SIGKILL).
  • invalid ownership metadata or an SSA conflict? The object does not belong to this release, or a field belongs to another field manager. managedFields shows to whom; adopt via label/annotation or --take-ownership, take a field with --force-conflicts – or change the chart so that it no longer sets the field.

Enterprise Note

  • Serialise CI pipelines per release (concurrency group, lock) and set Helm’s --timeout shorter than the job timeout, so that Helm gives up by itself and writes failed instead of leaving pending-upgrade behind.
  • Before anyone touches release Secrets: back up all sh.helm.release.v1.* Secrets of the namespace (e.g. with a backup tool), four-eyes principle, ticket.
  • Prevent manual scaling and kubectl edit on Helm objects via RBAC or policy; where an HPA scales, the chart renders no replicas.
  • Prepare Kubernetes upgrades with a list of removed APIs and move all releases to new APIs beforehand – rewriting release Secrets remains an emergency measure.
  • Monitoring on release status: alert when the current revision of a release is pending-* or failed for longer than the timeout – not on old Secrets.