Day 19: Troubleshooting
Goal: Assign every Helm error to a layer, resolve stuck and failed releases cleanly, and fix the typical API errors without losing data. · Duration: approx. 90 min · Prerequisite: Day 18
Briefing
- Three layers: rendering (errors with file and line, no cluster needed), API (the server rejects), runtime (everything accepted, but the Pods do not run).
- The first minute changes nothing:
status,history,get values,get manifest, events. - Helm locks a release through the revision with status
pending-*. If Helm is killed hard, the lock stays. - Many Helm errors are Kubernetes errors: immutable fields, foreign objects, field manager conflicts, removed APIs.
- Back up first, then repair – release Secrets above all.
Drill 1: Three layers
All experiments run in tag19, with helper
and small throwaway charts.
# on prod
mkdir -p ~/helm/tag19 && cd ~/helm/charts
kubectl create namespace tag19
helm template t ./notes 2>&1 | head -1
helm install a ./helper -n tag19 --set port=abc 2>&1 | cut -c1-80
helm list -n tag19 --failed -q
kubectl -n tag19 get svc,deploy -l app.kubernetes.io/instance=a -o name
helm template a ./helper --set port=abc | kubectl apply --dry-run=server -f - 2>&1 \
| cut -c1-80
helm install l ./helper -n tag19 --set image.tag=2.99.0 | grep STATUS
sleep 10
kubectl -n tag19 get pods -l app.kubernetes.io/instance=lExpected (abridged):
namespace/tag19 created
Error: execution error at (notes/templates/deployment.yaml:35:27): apiKey.existingSe…
Error: INSTALLATION FAILED: server-side apply failed for object tag19/a-helper a
a
service/a-helper
service/a-helper created (server dry run)
Error from server (BadRequest): error when creating "STDIN": Deployment in versi
STATUS: deployed
NAME READY STATUS RESTARTS AGE
l-helper-8bcc4b48c-r4qz4 0/1 ErrImagePull 0 10s
The first error is the required check of
notes (apiKey.existingSecret fehlt,
‘missing’), which fires because no values file is given.
| Layer | Sign | Tool |
|---|---|---|
| Rendering | execution error at (file:line) |
helm template --debug |
| API | apply failed, invalid,
conflict |
kubectl apply --dry-run=server |
| Runtime | deployed, but Pods not ready |
kubectl get/describe, events, logs |
The failed install a leaves behind a failed
release and a Service – Helm does not clean up
half-created objects. helm install --dry-run=server would
not have found the error (Day 4),
kubectl apply --dry-run=server does.
Save your progress? Sign up for free
Drill 2: The first minute
Release l reports deployed and still does
not run. Five sources, no changes:
# on prod
helm status l -n tag19 | grep -E '^(STATUS|REVISION|DESCRIPTION)'
helm status l -n tag19 | grep -A2 'v1/Pod'
helm history l -n tag19
helm get values l -n tag19
helm get values l -n tag19 --all | head -4
helm get manifest l -n tag19 | grep 'image:'
kubectl -n tag19 get events --field-selector reason=Failed \
-o jsonpath='{.items[0].message}' | fold -w 88; echoExpected (abridged):
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
l-helper-8bcc4b48c-r4qz4 0/1 ErrImagePull 0 11s
REVISION UPDATED STATUS CHART ... DESCRIPTION
1 Fri Nov 20 07:25:52 2026 deployed helper-0.2.1 ... Install complete
USER-SUPPLIED VALUES:
image:
tag: 2.99.0
COMPUTED VALUES:
image:
repository: ghcr.io/mccutchen/go-httpbin
tag: 2.99.0
image: "ghcr.io/mccutchen/go-httpbin:2.99.0"
Failed to pull image "ghcr.io/mccutchen/go-httpbin:2.99.0": rpc error: code = NotFound d
esc = failed to pull and unpack image "ghcr.io/mccutchen/go-httpbin:2.99.0": failed to r
esolve reference "ghcr.io/mccutchen/go-httpbin:2.99.0": ghcr.io/mccutchen/go-httpbin:2.9
9.0: not found
In Helm 4, status also shows the Pods.
history names the last good revision and often already the
reason in DESCRIPTION. get values without
--all shows only what someone set – here a tag that does
not exist. This is a runtime error: Helm did everything right.
Save your progress? Sign up for free
Drill 3: Stuck upgrade – pending-upgrade
Repair first, then start an upgrade with --wait and kill
Helm hard – like a CI runner that ends with SIGKILL:
# on prod
helm upgrade l ./helper -n tag19 --set image.tag=2.26.0 --wait | grep REVISION
helm upgrade l ./helper -n tag19 --set image.tag=2.98.0 --wait --timeout 5m \
>/dev/null 2>&1 &
sleep 4; kill -9 $!
helm history l -n tag19 --max 2
helm upgrade l ./helper -n tag19 --set image.tag=2.26.0 2>&1 | tail -1
pgrep -a helm || echo "kein helm-Prozess"
kubectl -n tag19 get deploy l-helper \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'kein helm-Prozess means ‘no helm process’.
Expected (abridged):
REVISION: 2
REVISION UPDATED STATUS ... DESCRIPTION
2 Fri Nov 20 07:26:19 2026 deployed ... Upgrade complete
3 Fri Nov 20 07:26:23 2026 pending-upgrade ... Preparing upgrade
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
kein helm-Prozess
ghcr.io/mccutchen/go-httpbin:2.98.0
Revision 3 is the orphaned lock. The cluster has already taken the
broken image. Same order as in operations: is there really no Helm
running any more (not in another pipeline either)? How is the
application doing? Then the clean way – roll back to the last
deployed revision:
# on prod
helm rollback l 2 -n tag19 --wait --description "pending-upgrade aufgelöst"
helm history l -n tag19 --max 3
helm upgrade l ./helper -n tag19 --set image.tag=2.26.0 --wait | grep REVISIONThe description means ‘pending-upgrade resolved’.
Expected (abridged):
Rollback was a success! Happy Helming!
REVISION ... STATUS ... DESCRIPTION
2 ... superseded ... Upgrade complete
3 ... pending-upgrade ... Preparing upgrade
4 ... deployed ... pending-upgrade aufgelöst
REVISION: 5
Helm 4 allows the rollback despite the lock; revision 3 stays in the
history as pending-upgrade but no longer gets in the way.
The last resort, if the rollback does not work: delete exactly the
Secret of the stuck revision.
Warning: Delete only the newest
pending-*Secret, and back it up first. The label filter also finds the old, harmless revision 3 – so sort and double-check the name. If you delete all release Secrets, you delete Helm’s memory: the release is unknown afterwards, and the objects keep running without an owner.
# on prod
helm upgrade l ./helper -n tag19 --set image.tag=2.98.0 --wait --timeout 5m \
>/dev/null 2>&1 &
sleep 4; kill -9 $!
kubectl -n tag19 get secret -l owner=helm,name=l,status=pending-upgrade \
--sort-by=.metadata.creationTimestamp -o name
kubectl -n tag19 get secret sh.helm.release.v1.l.v6 -o yaml > ~/helm/tag19/l-v6.yaml
kubectl -n tag19 delete secret sh.helm.release.v1.l.v6
helm history l -n tag19 --max 1
kubectl -n tag19 get deploy l-helper \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
helm upgrade l ./helper -n tag19 --set image.tag=2.26.0 --wait | grep REVISIONExpected (abridged):
secret/sh.helm.release.v1.l.v3
secret/sh.helm.release.v1.l.v6
secret "sh.helm.release.v1.l.v6" deleted from tag19 namespace
REVISION ... STATUS ... DESCRIPTION
5 ... deployed ... Upgrade complete
ghcr.io/mccutchen/go-httpbin:2.98.0
REVISION: 6
Deleting the Secret changes nothing in the cluster – the Deployment
still runs 2.98.0; only the next upgrade repairs it. Prevention: only
one pipeline per release at a time, and a --timeout shorter
than the CI job timeout.
Save your progress? Sign up for free
Drill 4: Failed first install
The same hard abort, this time during the install:
# on prod
helm install pi ./helper -n tag19 --set image.tag=2.98.0 --wait --timeout 5m \
>/dev/null 2>&1 &
sleep 4; kill -9 $!
helm history pi -n tag19
helm upgrade --install pi ./helper -n tag19 2>&1 | tail -1
helm rollback pi -n tag19 2>&1 | tail -1
helm uninstall pi -n tag19Expected (abridged):
REVISION UPDATED STATUS ... DESCRIPTION
1 Fri Nov 20 07:26:39 2026 pending-install ... Initial install underway
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Error: release has no 0 version
release "pi" uninstalled
There is no good revision a rollback could lead to –
uninstall and install again (check first whether data is
affected). Release a from Drill 1 is different; its only
revision is failed:
# on prod
helm history a -n tag19 | cut -c1-88
helm upgrade a ./helper -n tag19 2>&1 | grep -o 'containerPort="abc"' | head -1
helm get values a -n tag19
helm upgrade a ./helper -n tag19 --reset-values --wait | grep -E '^(STATUS|REVISION)'
helm history a -n tag19 -o json | jq -r '.[] | "\(.revision) \(.status)"'Expected:
REVISION UPDATED STATUS CHART ... DESCRIPTION
1 Fri Nov 20 07:25:50 2026 failed helper-0.2.1 ... Release "a" failed: se
containerPort="abc"
USER-SUPPLIED VALUES:
port: abc
STATUS: deployed
REVISION: 3
1 failed
2 superseded
3 deployed
The message has no deployed releases from Helm 3 days
does not appear here in Helm 4.3.2: an upgrade of a release that has
only failed revisions is allowed. The trap is elsewhere: an upgrade
without values takes over the values of the last deployed
revision (Day 3) – if there is none, those of the failed one (Day 20,
round 4). The broken port comes back with it; only
--reset-values helps.
Save your progress? Sign up for free
Drill 5: field is immutable
A StatefulSet with data, and then the PVC is supposed to grow:
# on prod
kubectl -n tag19 create secret generic pg-db --from-literal=password=lab-db-123
helm install pg ./postgres -n tag19 --set auth.existingSecret=pg-db --wait | grep STATUS
kubectl -n tag19 exec pg-postgres-0 -- psql -U notes -qc \
'create table t (x int); insert into t values (19)'
helm upgrade pg ./postgres -n tag19 --reuse-values --set storage=2Gi 2>&1 \
| grep -o 'spec.volumeClaimTemplates: Invalid value\|field is immutable' | sort -uExpected:
secret/pg-db created
STATUS: deployed
field is immutable
spec.volumeClaimTemplates: Invalid value
volumeClaimTemplates, selector and
serviceName of a StatefulSet are immutable. The way out:
delete the StatefulSet, but leave the Pods and PVCs in place.
Warning:
kubectl delete statefulset --cascade=orphankeeps Pods and PVCs running. Without--cascade=orphan, Kubernetes deletes the Pods – the database is gone until the upgrade starts it again.
# on prod
kubectl -n tag19 delete statefulset pg-postgres --cascade=orphan
kubectl -n tag19 get pods -l app.kubernetes.io/instance=pg
helm upgrade pg ./postgres -n tag19 --reuse-values --set storage=2Gi --wait \
| grep REVISION
kubectl -n tag19 get pods -l app.kubernetes.io/instance=pg
kubectl -n tag19 get sts pg-postgres \
-o jsonpath='{.spec.volumeClaimTemplates[0].spec.resources.requests.storage}{"\n"}'
kubectl -n tag19 get pvc data-pg-postgres-0 \
-o jsonpath='{.status.capacity.storage}{"\n"}'
kubectl -n tag19 exec pg-postgres-0 -- psql -U notes -tAc 'select x from t'Expected:
statefulset.apps "pg-postgres" deleted from tag19 namespace
NAME READY STATUS RESTARTS AGE
pg-postgres-0 1/1 Running 0 22s
REVISION: 3
NAME READY STATUS RESTARTS AGE
pg-postgres-0 1/1 Running 0 24s
2Gi
1Gi
19
The new StatefulSet adopts the running Pod (same age, no restart),
and the data stays. But: the existing PVC stays at 1Gi –
volumeClaimTemplates applies only to new PVCs. You can grow
it only on the PVC itself, and only if the StorageClass supports
allowVolumeExpansion (local-path does not).
Second classic: a Job as a normal object whose command changes.
Warning:
--force-replaceis not a normal upgrade flag. It overwrites objects completely (including fields set by controllers) and switches the release to client-side apply (Day 14).
In the throwaway chart, sekunden means ‘seconds’,
einmal ‘once’ and schlafe ‘sleep’:
# on prod
cd ~/helm/tag19 && mkdir -p jobchart/templates
printf 'apiVersion: v2\nname: jobchart\nversion: 0.1.0\n' > jobchart/Chart.yaml
printf 'sekunden: "1"\n' > jobchart/values.yaml
cat > jobchart/templates/job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-einmal
spec:
template:
spec:
restartPolicy: Never
containers:
- name: schlafe
image: curlimages/curl:8.23.0
command: ["sleep", {{ .Values.sekunden | quote }}]
EOF
helm install j ./jobchart -n tag19 --wait --wait-for-jobs | grep STATUS
helm upgrade j ./jobchart -n tag19 --set-string sekunden=2 2>&1 | tail -1 | cut -c1-88
helm upgrade j ./jobchart -n tag19 --set-string sekunden=2 --force-replace 2>&1 \
| tail -1
helm upgrade j ./jobchart -n tag19 --set-string sekunden=2 --force-replace \
--server-side=false 2>&1 | tail -1 | cut -c1-88
kubectl -n tag19 delete job j-einmal
helm upgrade j ./jobchart -n tag19 --set-string sekunden=2 --wait --wait-for-jobs \
| grep REVISION
helm get metadata j -n tag19 | grep APPLY_METHOD
helm upgrade j ./jobchart -n tag19 --server-side=true --reuse-values | grep REVISION
helm get metadata j -n tag19 | grep APPLY_METHODExpected:
STATUS: deployed
Error: UPGRADE FAILED: server-side apply failed for object tag19/j-einmal batch/v1, Kind
Error: UPGRADE FAILED: invalid operation: cannot use server-side apply and force replace
together
Error: UPGRADE FAILED: failed to replace object: Job.batch "j-einmal" is invalid: [spec.
job.batch "j-einmal" deleted from tag19 namespace
REVISION: 5
APPLY_METHOD: client-side apply
REVISION: 6
APPLY_METHOD: server-side apply
Three findings. --force-replace (formerly
--force) does not work together with Server-Side Apply.
With --server-side=false, Helm replaces the object with a
PUT – it does not delete and recreate it, so immutable
fields stay immutable. And: the failed attempt with
--server-side=false switched the release to client-side,
because auto takes over the method of the previous
revision; only --server-side=true switches back. For Jobs,
the clean solution is a hook with before-hook-creation (Day
11) or a name that contains the revision.
Save your progress? Sign up for free
Drill 6: Who owns the object?
A chart with fixed names – like many charts from the internet. The
names are German: besitz ‘ownership’,
gemeinsam ‘shared’, von ‘from’,
hand ‘by hand’, notes-leser ‘notes
reader’:
# on prod
cd ~/helm/tag19 && mkdir -p besitz/templates
printf 'apiVersion: v2\nname: besitz\nversion: 0.1.0\n' > besitz/Chart.yaml
cat > besitz/templates/objekte.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: gemeinsam
data:
von: {{ .Release.Name }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: notes-leser
rules:
- apiGroups: [""]
resources: [configmaps]
verbs: [get, list]
EOF
kubectl -n tag19 create configmap gemeinsam --from-literal=von=hand
helm install b1 ./besitz -n tag19 2>&1 | fold -w 88
helm list -n tag19 -q | grep -c b1Expected:
configmap/gemeinsam created
Error: INSTALLATION FAILED: unable to continue with install: ConfigMap "gemeinsam" in na
mespace "tag19" exists and cannot be imported into the current release: invalid ownershi
p metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be
set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": mus
t be set to "b1"; annotation validation error: missing key "meta.helm.sh/release-namespa
ce": must be set to "tag19"
0
Helm adopts only objects with the label
app.kubernetes.io/managed-by: Helm and the annotations
meta.helm.sh/release-name and -namespace. The
check runs beforehand – not even a release is created.
--take-ownership skips it:
# on prod
helm install b1 ./besitz -n tag19 --take-ownership 2>&1 | tail -1 | fold -w 88
kubectl -n tag19 get cm gemeinsam --show-managed-fields -o json \
| jq -c '.metadata.managedFields[] | {manager, operation, fields: .fieldsV1}'
helm upgrade b1 ./besitz -n tag19 --take-ownership --force-conflicts | grep REVISION
kubectl -n tag19 get cm gemeinsam -o jsonpath='{.data.von}{"\n"}'
kubectl -n tag19 get cm gemeinsam --show-managed-fields -o json \
| jq -r '.metadata.managedFields[].manager'Expected:
Error: INSTALLATION FAILED: conflict occurred while applying object tag19/gemeinsam /v1,
Kind=ConfigMap: Apply failed with 1 conflict: conflict with "kubectl-create" using v1:
.data.von
{"manager":"kubectl-create","operation":"Update","fields":{"f:data":{".":{},"f:von":…
REVISION: 2
b1
helm
kubectl-create
The ownership check is gone, and now Server-Side Apply speaks up (Day
14): the field data.von belongs to the field manager
kubectl-create, and Helm wants a different value. Who owns
what is in managedFields. --force-conflicts
hands the field to Helm. With cluster-wide objects, two releases
collide:
# on prod
kubectl create namespace tag19b
helm install b2 ./besitz -n tag19b 2>&1 | grep -o 'ClusterRole.*current value is "b1"' \
| fold -w 88
sed -i 's/name: notes-leser/name: {{ .Release.Name }}-notes-leser/' \
besitz/templates/objekte.yaml
kubectl -n tag19b create configmap gemeinsam --from-literal=von=b2
kubectl -n tag19b label configmap gemeinsam app.kubernetes.io/managed-by=Helm
kubectl -n tag19b annotate configmap gemeinsam meta.helm.sh/release-name=b2 \
meta.helm.sh/release-namespace=tag19b
helm install b2 ./besitz -n tag19b | grep STATUS
kubectl get clusterrole -l app.kubernetes.io/managed-by=Helm -o name | grep leserExpected:
namespace/tag19b created
ClusterRole "notes-leser" in namespace "" exists and cannot be imported into the current
release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/rel
ease-name" must equal "b2": current value is "b1"
configmap/gemeinsam created
configmap/gemeinsam labeled
configmap/gemeinsam annotated
STATUS: deployed
clusterrole.rbac.authorization.k8s.io/b2-notes-leser
clusterrole.rbac.authorization.k8s.io/notes-leser
The right repair is in the chart: cluster-wide objects get the
release name. Helm adopts the hand-made ConfigMap in tag19b
after label and annotate without a conflict –
it already has the value the chart sets.
Save your progress? Sign up for free
Drill 7: Conflict after kubectl scale
Someone scales by hand during the night. In the morning the next upgrade comes – without any change:
# on prod
cd ~/helm/charts
kubectl -n tag19 scale deployment l-helper --replicas=3
helm upgrade l ./helper -n tag19 --reuse-values 2>&1 | tail -1 | fold -w 88
kubectl -n tag19 get deploy l-helper --show-managed-fields -o json \
| jq -r '.metadata.managedFields[]
| select(.fieldsV1."f:spec"."f:replicas") | [.manager, .subresource] | @tsv'
helm upgrade l ./helper -n tag19 --reuse-values --set replicaCount=2 --force-conflicts \
--wait | grep REVISION
kubectl -n tag19 get deploy l-helper -o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n tag19 get deploy l-helper --show-managed-fields -o json \
| jq -r '.metadata.managedFields[]
| select(.fieldsV1."f:spec"."f:replicas") | [.manager, .operation] | @tsv'Expected:
deployment.apps/l-helper scaled
Error: UPGRADE FAILED: conflict occurred while applying object tag19/l-helper apps/v1, K
ind=Deployment: Apply failed with 1 conflict: conflict with "kubectl" with subresource "
scale" using apps/v1: .spec.replicas
kubectl scale
REVISION: 8
2
helm Apply
With Server-Side Apply (the default for new installs in Helm 4, Day
14), spec.replicas belongs to the manager
kubectl after kubectl scale. The chart always
renders replicas, so every upgrade fails. Either Helm takes
the field back with --force-conflicts, or the chart leaves
out replicas when an HPA or a human scales – with an
ownership handover when you switch, otherwise the Deployment briefly
drops to one Pod (Day 14, Drill 5). If you do not know this, all you see
is a pipeline that suddenly turns red.
helm diff upgrade --three-way-merge (Day 14) would have
shown the drift beforehand.
Save your progress? Sign up for free
Drill 8: Removed API in the stored release
After a Kubernetes upgrade, the stored manifest can
contain an API that the cluster no longer knows. On k3s 1.37 you can
only reproduce this by manipulating the release Secret
(policy/v1beta1 has been gone since 1.25).
kaputt means ‘broken’:
# on prod
cd ~/helm/tag19 && mkdir -p altapi/templates
printf 'apiVersion: v2\nname: altapi\nversion: 0.1.0\n' > altapi/Chart.yaml
cat > altapi/templates/pdb.yaml <<'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ .Release.Name }}-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/instance: {{ .Release.Name }}
EOF
helm install alt ./altapi -n tag19 | grep STATUS
S=sh.helm.release.v1.alt.v1
kubectl -n tag19 get secret $S -o jsonpath='{.data.release}' \
| base64 -d | base64 -d | gzip -d \
| jq '.manifest |= sub("policy/v1"; "policy/v1beta1")' \
| gzip -c | base64 -w0 | base64 -w0 > kaputt.b64
printf '{"data":{"release":"%s"}}' "$(cat kaputt.b64)" > kaputt.json
kubectl -n tag19 patch secret $S --type merge --patch-file kaputt.json
helm get manifest alt -n tag19 | grep apiVersion
helm upgrade alt ./altapi -n tag19 2>&1 | fold -w 88
helm history alt -n tag19 -o json | jq -r '.[] | "\(.revision) \(.status)"'Expected:
STATUS: deployed
secret/sh.helm.release.v1.alt.v1 patched
apiVersion: policy/v1beta1
Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest:
resource mapping not found for name: "alt-pdb" namespace: "" from "": no matches for ki
nd "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first
1 deployed
The chart is fine; Helm already fails while reading the old state –
and does not even create a failed revision. The plugin
helm-mapkubeapis rewrites such manifests automatically; it is not
installed in the lab, and whether version 0.6.1 runs reliably with Helm
4 is not established. You can find such releases beforehand with
pluto detect-helm (Day 17). By hand it is the same route
backwards (sicherung means ‘backup’, repariert
‘repaired’):
Warning: Rewriting a release Secret by hand is open-heart surgery. Back it up first (
-o yaml), touch only the latest revision, and keep the double Base64 encoding and gzip exactly. A patch file instead of-pavoids overlong command lines for large releases.
# on prod
kubectl -n tag19 get secret $S -o yaml > alt-v1-sicherung.yaml
kubectl -n tag19 get secret $S -o jsonpath='{.data.release}' \
| base64 -d | base64 -d | gzip -d \
| jq '.manifest |= gsub("policy/v1beta1"; "policy/v1")' \
| gzip -c | base64 -w0 | base64 -w0 > repariert.b64
printf '{"data":{"release":"%s"}}' "$(cat repariert.b64)" > repariert.json
kubectl -n tag19 patch secret $S --type merge --patch-file repariert.json
helm get manifest alt -n tag19 | grep apiVersion
helm upgrade alt ./altapi -n tag19 | grep -E '^(STATUS|REVISION)'Expected:
secret/sh.helm.release.v1.alt.v1 patched
apiVersion: policy/v1
STATUS: deployed
REVISION: 2
Prevention is cheaper: before every Kubernetes upgrade, render the
charts against the new version with
helm template --kube-version (Day 4), and bring the
releases to current APIs with an upgrade while the old cluster is still
running.
| Message | Cause | Drill |
|---|---|---|
another operation … is in progress |
orphaned pending-* revision |
3, 4 |
release has no 0 version |
rollback without a good revision | 4 |
field is immutable |
selector, volumeClaimTemplates, Job template |
5 |
invalid ownership metadata |
object belongs to no release / another release | 6 |
conflict with "…" using … |
another field manager owns the field | 6, 7 |
unable to build kubernetes objects … |
removed API in the stored release | 8 |
context deadline exceeded |
image, probe, hook Job, PVC | Challenge |
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all releases in the cluster whose current revision is
failedorpending-*. Target time: 15 s - Find all release Secrets in the cluster with status
pending-upgrade. Target time: 20 s - Which revision of
lfailed last, and with which values? Target time: 45 s - Which release owns the ConfigMap
gemeinsamintag19? Target time: 20 s
Solution
# on prod
helm list -A --failed --pending
kubectl get secret -A -l owner=helm,status=pending-upgrade -o name
helm history l -n tag19 -o json | jq -r '[.[] | select(.status=="failed")][-1].revision'
helm get values l -n tag19 --revision 7
kubectl -n tag19 get cm gemeinsam -o jsonpath='{.metadata.annotations}{"\n"}'Expected:
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
secret/sh.helm.release.v1.l.v3
7
USER-SUPPLIED VALUES:
image:
tag: 2.26.0
{"meta.helm.sh/release-name":"b1","meta.helm.sh/release-namespace":"tag19"}
helm list filters by the status of the
current revision – all releases are repaired, so the
list is empty. The Secret l.v3, on the other hand, is the
old lock from Drill 3: harmless, but an alert on “pending Secrets exist”
would fire falsely here. Revision 7 was the upgrade without changes that
failed because of kubectl scale.
Save your progress? Sign up for free
Checkpoint
# on prod
helm list -A --failed --pending | wc -lExpected: 1 (only the header line).
# on prod
helm list -n tag19 -o json | jq -r '.[] | "\(.name) \(.status)"' | sort | tr '\n' ' 'Expected:
a deployed alt deployed b1 deployed j deployed l deployed pg deployed
(after the Challenge also c deployed).
# on prod
helm status notes -n notes | grep -E '^STATUS'Expected: STATUS: deployed – the prod
release noticed none of this.
Challenge
A colleague sent off this command, and the pipeline reports
context deadline exceeded. Find the cause without looking
at the command, and bring c to deployed.
# on prod
cd ~/helm/charts
helm upgrade --install c ./helper -n tag19 --set probePath=/status/503 \
--wait --timeout 30s 2>&1 | tail -2Expected:
Error: resource Deployment/tag19/c-helper not ready. status: InProgress, message: Ava…
context deadline exceeded
Hint: context deadline exceeded only
means “Helm waited” – the answer is in the Pods and their events.
Solution
# on prod
helm status c -n tag19 | grep -E '^STATUS'
kubectl -n tag19 get pods -l app.kubernetes.io/instance=c
kubectl -n tag19 describe pod -l app.kubernetes.io/instance=c | grep -m1 'statuscode'
helm get values c -n tag19
helm upgrade c ./helper -n tag19 --reset-values --wait --timeout 60s | grep REVISION
helm history c -n tag19 -o json | jq -r '.[] | "\(.revision) \(.status)"'Expected (abridged):
STATUS: failed
NAME READY STATUS RESTARTS AGE
c-helper-7bd5bf5496-rbsm2 0/1 Running 0 31s
Warning Unhealthy 8s (x5 over 30s) kubelet spec.containers{helper}: Readiness
probe failed: HTTP probe failed with statuscode: 503
USER-SUPPLIED VALUES:
probePath: /status/503
REVISION: 2
1 superseded
2 deployed
The Pod runs but is never ready: the readinessProbe asks for
/status/503. The cause was in get values, the
proof in the events. --reset-values discards the wrong
value (an upgrade without values would have taken it over from the
failed revision, Drill 4). The other candidates for the same error:
image cannot be pulled (ErrImagePull, Drill 2), hook Job
stuck (kubectl get jobs, Day 11), PVC Pending
(kubectl get pvc, events of the PVC).
Save your progress? Sign up for free
Cleanup
# on prod
helm uninstall -n tag19 $(helm list -n tag19 -q)
helm uninstall b2 -n tag19b
kubectl delete namespace tag19 tag19b
kubectl get clusterrole -o name | grep -c notes-leserExpected (abridged):
release "a" uninstalled
…
release "pg" uninstalled
release "b2" uninstalled
namespace "tag19" deleted
namespace "tag19b" deleted
0
The PVC data-pg-postgres-0 survives
helm uninstall (Day 9) and only disappears with the
namespace. Nothing changed in the notes chart today, so
there is no commit.
Interview Note
- A Helm upgrade fails. How do you proceed? Determine the
layer (rendering, API, runtime), then
status,history,get values,get manifest, events and logs – still without changing anything. Back up first, then roll back or repair. another operation is in progress– what do you do? Check that really no run is active, check the state of the application, roll back to the lastdeployedrevision. Only if that does not work: back up and delete the Secret of the newestpendingrevision. Afterwards fix the cause in CI (lock, timeouts, no SIGKILL).invalid ownership metadataor an SSA conflict? The object does not belong to this release, or a field belongs to another field manager.managedFieldsshows to whom; adopt via label/annotation or--take-ownership, take a field with--force-conflicts– or change the chart so that it no longer sets the field.
Enterprise Note
- Serialise CI pipelines per release (concurrency group, lock) and set
Helm’s
--timeoutshorter than the job timeout, so that Helm gives up by itself and writesfailedinstead of leavingpending-upgradebehind. - Before anyone touches release Secrets: back up all
sh.helm.release.v1.*Secrets of the namespace (e.g. with a backup tool), four-eyes principle, ticket. - Prevent manual scaling and
kubectl editon Helm objects via RBAC or policy; where an HPA scales, the chart renders noreplicas. - Prepare Kubernetes upgrades with a list of removed APIs and move all releases to new APIs beforehand – rewriting release Secrets remains an emergency measure.
- Monitoring on release status: alert when the
current revision of a release is
pending-*orfailedfor longer than the timeout – not on old Secrets.