Day 17: Rollouts, autoscaling, Jobs and Helm
Goal: You recognize stuck rollouts, a blind HPA, a stuck Helm release, a Job that gives up and configuration that does not arrive – and for each, you know which command tells the truth. · Duration: approx. 75 min · Prerequisite: Day 16
Briefing
- A Deployment rollout does not end with an error but with a
condition:
Progressing=False, reasonProgressDeadlineExceeded. - The HPA calculates in percent of the requests; no request, no target.
- Helm stores every release state as a Secret. If Helm dies in the
middle of an upgrade, the state stays
pending-upgrade– and blocks every further upgrade. - ConfigMap changes only reach volumes without
subPath, never environment variables.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Rollout stuck | exceeded its progress deadline |
Conditions, ReplicaSets |
| HPA blind | cpu: <unknown>/50% |
Events on the HPA |
| Helm blocked | another operation … in progress |
helm history |
| Job gives up | BackoffLimitExceeded |
Job conditions, logs |
| Configuration stale | Pod reads old value | Env vs. subPath vs. volume |
Drill 1: The rollout is stuck
Break it: Three web Pods; in the lab,
the progress deadline is shortened to 30 seconds (default 600). Then an
image tag that does not exist:
# on cp1
mkdir -p ~/debug/tag17 && cd ~/debug/tag17
kubectl create namespace tag17
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag17}
spec:
replicas: 3
progressDeadlineSeconds: 30
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
containers:
- name: web
image: traefik/whoami:v1.11
EOF
kubectl apply -f web.yaml
kubectl -n tag17 rollout status deploy/web
kubectl -n tag17 set image deploy/web web=traefik/whoami:v1.99Expected (abridged):
namespace/tag17 created
deployment.apps/web created
deployment "web" successfully rolled out
deployment.apps/web image updated
Symptom:
# on cp1
kubectl -n tag17 rollout status deploy/web --timeout=90s; echo "rc=$?"
kubectl -n tag17 get deploy webExpected (abridged):
error: deployment "web" exceeded its progress deadline
rc=1
NAME READY UP-TO-DATE AVAILABLE AGE
web 3/3 1 3 34s
3/3 ready, but only one UP-TO-DATE – the
service is running, the update is not.
Investigate:
# on cp1
kubectl -n tag17 get rs
kubectl -n tag17 get pods
J='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl -n tag17 get deploy web -o jsonpath="$J"Expected:
NAME DESIRED CURRENT READY AGE
web-5ccbc58945 3 3 3 37s
web-6577c6f697 1 1 0 35s
NAME READY STATUS RESTARTS AGE
web-5ccbc58945-fmq5t 1/1 Running 0 37s
web-5ccbc58945-gbwk4 1/1 Running 0 37s
web-5ccbc58945-j9nq2 1/1 Running 0 37s
web-6577c6f697-fj6g4 0/1 ErrImagePull 0 35s
Available=True MinimumReplicasAvailable
Progressing=False ProgressDeadlineExceeded
Cause: The new ReplicaSet never gets a ready Pod
(Day 4). The default strategy (maxSurge 25 %,
maxUnavailable 25 % – rounded down to zero with three
replicas) starts one new Pod and removes no old one as long as the new
one is not ready. After progressDeadlineSeconds, the
controller sets ProgressDeadlineExceeded – it does
not abort, it only reports.
Fix:
# on cp1
kubectl -n tag17 rollout history deploy/web
kubectl -n tag17 rollout undo deploy/web 2>&1 | tail -1
kubectl -n tag17 rollout status deploy/webExpected (abridged):
deployment.apps/web
REVISION CHANGE-CAUSE
1 <none>
2 <none>
deployment.apps/web rolled back
deployment "web" successfully rolled out
In 1.37, rollout undo warns that the
kubectl apply annotation is not rolled back along with it.
If the fault came in via kubectl apply, fix the file
instead – otherwise the next apply brings it back.
Verify:
# on cp1
kubectl -n tag17 get deploy web
kubectl -n tag17 get deploy web \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Expected:
NAME READY UP-TO-DATE AVAILABLE AGE
web 3/3 3 3 45s
traefik/whoami:v1.11
Save your progress? Sign up for free
Drill 2: The HPA sees nothing
Break it:
# on cp1
kubectl -n tag17 autoscale deploy web --cpu=50% --min=2 --max=5Expected:
horizontalpodautoscaler.autoscaling/web autoscaled
Symptom (after about 45 seconds):
# on cp1
sleep 45
kubectl -n tag17 get hpaExpected:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web cpu: <unknown>/50% 2 5 3 49s
Investigate:
# on cp1
kubectl -n tag17 events --for hpa/web --types=Warning | tail -2 | cut -c1-200 \
| fold -w 86
kubectl top pods -n tag17 | head -2Expected (abridged):
6s Warning FailedGetResourceMetric HorizontalPodAutoscaler/w
eb failed to get cpu utilization: missing request for cpu in container web of Pod we
b-5ccbc58945-gbwk4
...
NAME CPU(cores) MEMORY(bytes)
web-5ccbc58945-fmq5t 0m 1Mi
Cause: 50% means 50 percent of the CPU
request. The container has no request, so there is
nothing to take 50 percent of. kubectl top, on the other
hand, returns values – the metrics-server (Day 16) is not the
problem.
Fix:
# on cp1
kubectl -n tag17 set resources deploy/web --requests=cpu=10m
kubectl -n tag17 rollout status deploy/webVerify (after about a minute):
# on cp1
sleep 60
kubectl -n tag17 get hpaExpected:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web cpu: 0%/50% 2 5 3 2m5s
REPLICAS stays at 3 for now, even though
MINPODS allows 2: the HPA waits five minutes (stabilization
window) before it scales down.
Save your progress? Sign up for free
Drill 3: Helm is stuck in pending-upgrade
Break it: A sample chart, installed and then killed hard in the middle of an upgrade – just like an aborted CI job:
Warning:
kill -9terminates Helm without cleaning up. Lab only.
# on cp1
helm create demo
helm install demo ./demo -n tag17 --wait --timeout 2m | grep -E 'STATUS|REVISION'
helm upgrade demo ./demo -n tag17 --set image.tag=gibtsnicht --wait --timeout 5m &
sleep 5
kill -9 %1Expected (abridged):
Creating demo
STATUS: deployed
REVISION: 1
After kill -9, the shell reports that the background job
was killed (Killed helm upgrade demo …).
Symptom:
# on cp1
helm upgrade demo ./demo -n tag17 2>&1 | tail -1Expected:
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Investigate:
# on cp1
helm -n tag17 history demo -o json \
| jq -r '.[] | "\(.revision) \(.status) \(.description)"'
helm list -n tag17 -o json | jq -r '.[] | "\(.name) \(.revision) \(.status)"'
kubectl -n tag17 get secret -l owner=helmExpected:
1 deployed Install complete
2 pending-upgrade Preparing upgrade
demo 2 pending-upgrade
NAME TYPE DATA AGE
sh.helm.release.v1.demo.v1 helm.sh/release.v1 1 14s
sh.helm.release.v1.demo.v2 helm.sh/release.v1 1 10s
Cause: At the start of the upgrade, Helm writes
revision 2 as pending-upgrade, and only at the end
deployed or failed. kill -9
leaves the intermediate state behind, and every further upgrade believes
another one is still running. Helm 4 no longer knows
helm list -a; list shows all states.
Fix: Go back to the last good revision:
# on cp1
helm -n tag17 rollback demo 1 2>&1 | tail -1
helm -n tag17 history demo -o json \
| jq -r '.[] | "\(.revision) \(.status) \(.description)"'Expected:
Rollback was a success! Happy Helming!
1 superseded Install complete
2 pending-upgrade Preparing upgrade
3 deployed Rollback to 1
Revision 2 stays in the history as pending-upgrade –
what counts is the newest one.
Verify:
# on cp1
helm upgrade demo ./demo -n tag17 --wait | grep -E 'STATUS|REVISION'Expected:
STATUS: deployed
REVISION: 4
Save your progress? Sign up for free
Drill 4: The Job gives up
Break it: An import that fails with exit code 3 every time:
# on cp1
cat > job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata: {name: import, namespace: tag17}
spec:
backoffLimit: 3
template:
spec:
restartPolicy: Never
containers:
- name: import
image: busybox:1.37
command: [sh, -c, "echo Datei fehlt; exit 3"]
EOF
kubectl apply -f job.yamlSymptom:
# on cp1
T0=$(date +%s)
kubectl -n tag17 wait --for=condition=Failed job/import --timeout=180s
echo "nach $(( $(date +%s) - T0 )) s"
kubectl -n tag17 get job import
kubectl -n tag17 get pods -l job-name=importExpected:
job.batch/import condition met
nach 75 s
NAME STATUS COMPLETIONS DURATION AGE
import Failed 0/1 78s 78s
NAME READY STATUS RESTARTS AGE
import-2jv7f 0/1 Error 0 5s
import-4x2vx 0/1 Error 0 67s
import-j6pql 0/1 Error 0 79s
import-vtrqf 0/1 Error 0 46s
(nach 75 s means ‘after 75 s’.)
Investigate:
# on cp1
J='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
kubectl -n tag17 get job import -o jsonpath="$J"
kubectl -n tag17 logs job/importExpected:
FailureTarget=True BackoffLimitExceeded: Job has reached the specified backoff limit
Failed=True BackoffLimitExceeded: Job has reached the specified backoff limit
Found 4 pods, using pod/import-j6pql
Datei fehlt
Cause: One attempt plus three retries
(backoffLimit: 3), with growing pauses (10 s, 20 s, 40 s) –
75 seconds for a fault that was clear the first time. Here, exit code 3
means “input missing” (the log says Datei fehlt, ‘file
missing’); retrying never helps.
Fix: A podFailurePolicy makes the Job
fail immediately on exactly this exit code:
# on cp1
kubectl -n tag17 delete job import
cat > job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata: {name: import, namespace: tag17}
spec:
backoffLimit: 3
podFailurePolicy:
rules:
- action: FailJob
onExitCodes: {containerName: import, operator: In, values: [3]}
template:
spec:
restartPolicy: Never
containers:
- name: import
image: busybox:1.37
command: [sh, -c, "echo Datei fehlt; exit 3"]
EOF
kubectl apply -f job.yamlVerify:
# on cp1
T0=$(date +%s)
kubectl -n tag17 wait --for=condition=Failed job/import --timeout=180s
echo "nach $(( $(date +%s) - T0 )) s"
kubectl -n tag17 get job import -o jsonpath='{.status.conditions[-1].message}{"\n"}' \
| fold -w 86Expected:
job.batch/import condition met
nach 2 s
Container import for pod tag17/import-6xtnh failed with exit code 3 matching FailJob r
ule at index 0
One Pod instead of four, and the message names the reason. With
action: Ignore and
onPodConditions: [{type: DisruptionTarget}], terminations
caused by eviction or preemption do not count against
backoffLimit at all.
Save your progress? Sign up for free
Drill 5: The configuration does not arrive
Break it: A Pod reads the same ConfigMap key in
three ways – as an environment variable, as a single file via
subPath and as a volume:
# on cp1
cat > konfig.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: app-konfig, namespace: tag17}
data:
farbe: blau
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: leser, namespace: tag17}
spec:
replicas: 1
selector: {matchLabels: {app: leser}}
template:
metadata: {labels: {app: leser}}
spec:
containers:
- name: leser
image: busybox:1.37
command: [sleep, infinity]
env:
- name: FARBE
valueFrom: {configMapKeyRef: {name: app-konfig, key: farbe}}
volumeMounts:
- {name: konfig, mountPath: /etc/app/farbe, subPath: farbe}
- {name: konfig, mountPath: /konfig}
volumes:
- name: konfig
configMap: {name: app-konfig}
EOF
kubectl apply -f konfig.yaml
kubectl -n tag17 rollout status deploy/leser
kubectl -n tag17 patch configmap app-konfig -p '{"data":{"farbe":"rot"}}'The key farbe (‘color’) changes from blau
(‘blue’) to rot (‘red’); leser means
‘reader’.
Symptom (after about a minute):
# on cp1
sleep 60
kubectl -n tag17 exec deploy/leser -- \
sh -c 'echo env=$FARBE subPath=$(cat /etc/app/farbe) volume=$(cat /konfig/farbe)'Expected:
env=blau subPath=blau volume=rot
Cause: Environment variables are only set at
startup. A subPath mount is a single file fixed at startup
– the kubelet never replaces it. Only the volume without
subPath follows the ConfigMap (via a symlink, after up to
about a minute).
Fix: Roll out again – or have the application read from the volume directory:
# on cp1
kubectl -n tag17 rollout restart deploy/leser
kubectl -n tag17 rollout status deploy/leserVerify:
# on cp1
kubectl -n tag17 exec deploy/leser -- \
sh -c 'echo env=$FARBE subPath=$(cat /etc/app/farbe) volume=$(cat /konfig/farbe)'Expected:
env=rot subPath=rot volume=rot
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the conditions of the Deployment
webwith their reason. Target time: 30 s - Show all Helm releases in the cluster that are not
deployed. Target time: 20 s - Show all failed Jobs in the cluster. Target time: 30 s
Solution
# on cp1
J='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl -n tag17 get deploy web -o jsonpath="$J"
helm list -A --pending --failed -q; echo "rc=$?"
kubectl get jobs -A | grep -E 'NAME|Failed' | tr -s ' 'Expected:
Available=True MinimumReplicasAvailable
Progressing=True NewReplicaSetAvailable
rc=0
NAMESPACE NAME STATUS COMPLETIONS DURATION AGE
tag17 import Failed 0/1 89s 89s
-q only prints names – none here, the Helm release is
clean after Drill 3.
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag17 get deploy
helm -n tag17 status demo -o json | jq -r .info.statusExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
demo 1/1 1 1 3m25s
leser 1/1 1 1 84s
web 2/2 2 2 6m22s
deployed
web is now at 2 – the HPA’s stabilization window has
expired.
Challenge
The Deployment api gets a new readiness probe. For two
minutes, the rollout has reported nothing new, there are no errors, and
the service is running. Find out what the rollout is stuck on and bring
it into a clean state.
# on cp1
cat > api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api, namespace: tag17}
spec:
replicas: 3
strategy:
rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
selector: {matchLabels: {app: api}}
template:
metadata: {labels: {app: api}}
spec:
containers:
- name: api
image: traefik/whoami:v1.11
readinessProbe:
httpGet: {path: /, port: 80}
EOF
kubectl apply -f api.yaml
kubectl -n tag17 rollout status deploy/api
P=/spec/template/spec/containers/0/readinessProbe/httpGet/port
kubectl -n tag17 patch deploy api --type=json \
-p '[{"op": "replace", "path": "'$P'", "value": 8080}]'Hint: With the default deadline of 600 seconds,
ProgressDeadlineExceeded only shows up after ten minutes –
look at the ReplicaSets.
Solution
# on cp1
sleep 40
kubectl -n tag17 rollout status deploy/api --timeout=5s
kubectl -n tag17 get rs -l app=api
kubectl -n tag17 get pods -l app=api
kubectl -n tag17 events --types=Warning | grep 'Pod/api' | grep -m1 Unhealthy \
| cut -c1-200 | fold -w 86Expected:
Waiting for deployment "api" rollout to finish: 1 out of 3 new replicas have been upda
ted...
error: timed out waiting for the condition
NAME DESIRED CURRENT READY AGE
api-6745d6c768 1 1 0 48s
api-77d7d744d8 3 3 3 51s
NAME READY STATUS RESTARTS AGE
api-6745d6c768-dtp5c 0/1 Running 0 49s
api-77d7d744d8-c2f8c 1/1 Running 0 52s
api-77d7d744d8-hxdzb 1/1 Running 0 52s
api-77d7d744d8-rqq4m 1/1 Running 0 52s
7s (x7 over 47s) Warning Unhealthy Pod/api-6745d6c768-dt
p5c Readiness probe failed: Get "http://10.42.0.168:8080/": dial tcp 10.42.0.168:
8080: connect: connection re
The new Pod is running but never becomes ready – the probe asks port
8080, whoami listens on 80. With
maxUnavailable: 0, all three old Pods stay: no outage, but
no progress either. Go back to the last good version:
# on cp1
kubectl -n tag17 rollout undo deploy/api 2>&1 | tail -1
kubectl -n tag17 rollout status deploy/api
kubectl -n tag17 get rs -l app=apiExpected (abridged):
deployment.apps/api rolled back
deployment "api" successfully rolled out
NAME DESIRED CURRENT READY AGE
api-6745d6c768 0 0 0 53s
api-77d7d744d8 3 3 3 56s
Save your progress? Sign up for free
Cleanup
# on cp1
helm uninstall demo -n tag17
kubectl delete namespace tag17
helm list -A | grep -c demoExpected:
release "demo" uninstalled
namespace "tag17" deleted
0
Interview Note
- Rollout stuck – how do I see it?
rollout statusaborts withexceeded its progress deadline;get rsshows the new ReplicaSet without ready Pods. The controller does not roll back by itself. - HPA shows
<unknown>? Events on the HPA: missing request or missing metrics API (kubectl get apiservice, Day 16). - Helm
another operation is in progress?helm historyshowspending-*;helm rollbackto the last good revision, never delete the release Secrets by hand.
Enterprise Note
- Choose
progressDeadlineSecondsshort enough that CI pipelines fail withrollout statusinstead of waiting an hour; automatic rollback belongs in the pipeline or in Argo Rollouts/Flagger. - Always run Helm in CI with
--rollback-on-failure(in Helm 3:--atomic) and a timeout; prevent parallel pipelines on the same release. - Enforce requests for all containers (LimitRange, Kyverno), otherwise every HPA is blind.
- Deliver configuration via volumes without
subPathor with a hash in the Pod template (checksum annotation in the Helm chart), so that changes trigger a rollout.