Skip to content
Hands-on IT IT Drills

Day 17: Rollouts, autoscaling, Jobs and Helm

Goal: You recognize stuck rollouts, a blind HPA, a stuck Helm release, a Job that gives up and configuration that does not arrive – and for each, you know which command tells the truth. · Duration: approx. 75 min · Prerequisite: Day 16

Briefing

  • A Deployment rollout does not end with an error but with a condition: Progressing=False, reason ProgressDeadlineExceeded.
  • The HPA calculates in percent of the requests; no request, no target.
  • Helm stores every release state as a Secret. If Helm dies in the middle of an upgrade, the state stays pending-upgrade – and blocks every further upgrade.
  • ConfigMap changes only reach volumes without subPath, never environment variables.
Fault Symptom Where the answer is
Rollout stuck exceeded its progress deadline Conditions, ReplicaSets
HPA blind cpu: <unknown>/50% Events on the HPA
Helm blocked another operation … in progress helm history
Job gives up BackoffLimitExceeded Job conditions, logs
Configuration stale Pod reads old value Env vs. subPath vs. volume

Drill 1: The rollout is stuck

Break it: Three web Pods; in the lab, the progress deadline is shortened to 30 seconds (default 600). Then an image tag that does not exist:

# on cp1
mkdir -p ~/debug/tag17 && cd ~/debug/tag17
kubectl create namespace tag17
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag17}
spec:
  replicas: 3
  progressDeadlineSeconds: 30
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: traefik/whoami:v1.11
EOF
kubectl apply -f web.yaml
kubectl -n tag17 rollout status deploy/web
kubectl -n tag17 set image deploy/web web=traefik/whoami:v1.99

Expected (abridged):

namespace/tag17 created
deployment.apps/web created
deployment "web" successfully rolled out
deployment.apps/web image updated

Symptom:

# on cp1
kubectl -n tag17 rollout status deploy/web --timeout=90s; echo "rc=$?"
kubectl -n tag17 get deploy web

Expected (abridged):

error: deployment "web" exceeded its progress deadline
rc=1
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    3/3     1            3           34s

3/3 ready, but only one UP-TO-DATE – the service is running, the update is not.

Investigate:

# on cp1
kubectl -n tag17 get rs
kubectl -n tag17 get pods
J='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl -n tag17 get deploy web -o jsonpath="$J"

Expected:

NAME             DESIRED   CURRENT   READY   AGE
web-5ccbc58945   3         3         3       37s
web-6577c6f697   1         1         0       35s
NAME                   READY   STATUS         RESTARTS   AGE
web-5ccbc58945-fmq5t   1/1     Running        0          37s
web-5ccbc58945-gbwk4   1/1     Running        0          37s
web-5ccbc58945-j9nq2   1/1     Running        0          37s
web-6577c6f697-fj6g4   0/1     ErrImagePull   0          35s
Available=True MinimumReplicasAvailable
Progressing=False ProgressDeadlineExceeded

Cause: The new ReplicaSet never gets a ready Pod (Day 4). The default strategy (maxSurge 25 %, maxUnavailable 25 % – rounded down to zero with three replicas) starts one new Pod and removes no old one as long as the new one is not ready. After progressDeadlineSeconds, the controller sets ProgressDeadlineExceeded – it does not abort, it only reports.

Fix:

# on cp1
kubectl -n tag17 rollout history deploy/web
kubectl -n tag17 rollout undo deploy/web 2>&1 | tail -1
kubectl -n tag17 rollout status deploy/web

Expected (abridged):

deployment.apps/web
REVISION  CHANGE-CAUSE
1         <none>
2         <none>
deployment.apps/web rolled back
deployment "web" successfully rolled out

In 1.37, rollout undo warns that the kubectl apply annotation is not rolled back along with it. If the fault came in via kubectl apply, fix the file instead – otherwise the next apply brings it back.

Verify:

# on cp1
kubectl -n tag17 get deploy web
kubectl -n tag17 get deploy web \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    3/3     3            3           45s
traefik/whoami:v1.11

Save your progress? Sign up for free

Drill 2: The HPA sees nothing

Break it:

# on cp1
kubectl -n tag17 autoscale deploy web --cpu=50% --min=2 --max=5

Expected:

horizontalpodautoscaler.autoscaling/web autoscaled

Symptom (after about 45 seconds):

# on cp1
sleep 45
kubectl -n tag17 get hpa

Expected:

NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: <unknown>/50%   2         5         3          49s

Investigate:

# on cp1
kubectl -n tag17 events --for hpa/web --types=Warning | tail -2 | cut -c1-200 \
  | fold -w 86
kubectl top pods -n tag17 | head -2

Expected (abridged):

6s                  Warning   FailedGetResourceMetric        HorizontalPodAutoscaler/w
eb   failed to get cpu utilization: missing request for cpu in container web of Pod we
b-5ccbc58945-gbwk4
...
NAME                   CPU(cores)   MEMORY(bytes)
web-5ccbc58945-fmq5t   0m           1Mi

Cause: 50% means 50 percent of the CPU request. The container has no request, so there is nothing to take 50 percent of. kubectl top, on the other hand, returns values – the metrics-server (Day 16) is not the problem.

Fix:

# on cp1
kubectl -n tag17 set resources deploy/web --requests=cpu=10m
kubectl -n tag17 rollout status deploy/web

Verify (after about a minute):

# on cp1
sleep 60
kubectl -n tag17 get hpa

Expected:

NAME   REFERENCE        TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: 0%/50%   2         5         3          2m5s

REPLICAS stays at 3 for now, even though MINPODS allows 2: the HPA waits five minutes (stabilization window) before it scales down.

Save your progress? Sign up for free

Drill 3: Helm is stuck in pending-upgrade

Break it: A sample chart, installed and then killed hard in the middle of an upgrade – just like an aborted CI job:

Warning: kill -9 terminates Helm without cleaning up. Lab only.

# on cp1
helm create demo
helm install demo ./demo -n tag17 --wait --timeout 2m | grep -E 'STATUS|REVISION'
helm upgrade demo ./demo -n tag17 --set image.tag=gibtsnicht --wait --timeout 5m &
sleep 5
kill -9 %1

Expected (abridged):

Creating demo
STATUS: deployed
REVISION: 1

After kill -9, the shell reports that the background job was killed (Killed helm upgrade demo …).

Symptom:

# on cp1
helm upgrade demo ./demo -n tag17 2>&1 | tail -1

Expected:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Investigate:

# on cp1
helm -n tag17 history demo -o json \
  | jq -r '.[] | "\(.revision) \(.status) \(.description)"'
helm list -n tag17 -o json | jq -r '.[] | "\(.name) \(.revision) \(.status)"'
kubectl -n tag17 get secret -l owner=helm

Expected:

1 deployed Install complete
2 pending-upgrade Preparing upgrade
demo 2 pending-upgrade
NAME                         TYPE                 DATA   AGE
sh.helm.release.v1.demo.v1   helm.sh/release.v1   1      14s
sh.helm.release.v1.demo.v2   helm.sh/release.v1   1      10s

Cause: At the start of the upgrade, Helm writes revision 2 as pending-upgrade, and only at the end deployed or failed. kill -9 leaves the intermediate state behind, and every further upgrade believes another one is still running. Helm 4 no longer knows helm list -a; list shows all states.

Fix: Go back to the last good revision:

# on cp1
helm -n tag17 rollback demo 1 2>&1 | tail -1
helm -n tag17 history demo -o json \
  | jq -r '.[] | "\(.revision) \(.status) \(.description)"'

Expected:

Rollback was a success! Happy Helming!
1 superseded Install complete
2 pending-upgrade Preparing upgrade
3 deployed Rollback to 1

Revision 2 stays in the history as pending-upgrade – what counts is the newest one.

Verify:

# on cp1
helm upgrade demo ./demo -n tag17 --wait | grep -E 'STATUS|REVISION'

Expected:

STATUS: deployed
REVISION: 4

Save your progress? Sign up for free

Drill 4: The Job gives up

Break it: An import that fails with exit code 3 every time:

# on cp1
cat > job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata: {name: import, namespace: tag17}
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: import
        image: busybox:1.37
        command: [sh, -c, "echo Datei fehlt; exit 3"]
EOF
kubectl apply -f job.yaml

Symptom:

# on cp1
T0=$(date +%s)
kubectl -n tag17 wait --for=condition=Failed job/import --timeout=180s
echo "nach $(( $(date +%s) - T0 )) s"
kubectl -n tag17 get job import
kubectl -n tag17 get pods -l job-name=import

Expected:

job.batch/import condition met
nach 75 s
NAME     STATUS   COMPLETIONS   DURATION   AGE
import   Failed   0/1           78s        78s
NAME           READY   STATUS   RESTARTS   AGE
import-2jv7f   0/1     Error    0          5s
import-4x2vx   0/1     Error    0          67s
import-j6pql   0/1     Error    0          79s
import-vtrqf   0/1     Error    0          46s

(nach 75 s means ‘after 75 s’.)

Investigate:

# on cp1
J='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
kubectl -n tag17 get job import -o jsonpath="$J"
kubectl -n tag17 logs job/import

Expected:

FailureTarget=True BackoffLimitExceeded: Job has reached the specified backoff limit
Failed=True BackoffLimitExceeded: Job has reached the specified backoff limit
Found 4 pods, using pod/import-j6pql
Datei fehlt

Cause: One attempt plus three retries (backoffLimit: 3), with growing pauses (10 s, 20 s, 40 s) – 75 seconds for a fault that was clear the first time. Here, exit code 3 means “input missing” (the log says Datei fehlt, ‘file missing’); retrying never helps.

Fix: A podFailurePolicy makes the Job fail immediately on exactly this exit code:

# on cp1
kubectl -n tag17 delete job import
cat > job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata: {name: import, namespace: tag17}
spec:
  backoffLimit: 3
  podFailurePolicy:
    rules:
    - action: FailJob
      onExitCodes: {containerName: import, operator: In, values: [3]}
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: import
        image: busybox:1.37
        command: [sh, -c, "echo Datei fehlt; exit 3"]
EOF
kubectl apply -f job.yaml

Verify:

# on cp1
T0=$(date +%s)
kubectl -n tag17 wait --for=condition=Failed job/import --timeout=180s
echo "nach $(( $(date +%s) - T0 )) s"
kubectl -n tag17 get job import -o jsonpath='{.status.conditions[-1].message}{"\n"}' \
  | fold -w 86

Expected:

job.batch/import condition met
nach 2 s
Container import for pod tag17/import-6xtnh failed with exit code 3 matching FailJob r
ule at index 0

One Pod instead of four, and the message names the reason. With action: Ignore and onPodConditions: [{type: DisruptionTarget}], terminations caused by eviction or preemption do not count against backoffLimit at all.

Save your progress? Sign up for free

Drill 5: The configuration does not arrive

Break it: A Pod reads the same ConfigMap key in three ways – as an environment variable, as a single file via subPath and as a volume:

# on cp1
cat > konfig.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: app-konfig, namespace: tag17}
data:
  farbe: blau
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: leser, namespace: tag17}
spec:
  replicas: 1
  selector: {matchLabels: {app: leser}}
  template:
    metadata: {labels: {app: leser}}
    spec:
      containers:
      - name: leser
        image: busybox:1.37
        command: [sleep, infinity]
        env:
        - name: FARBE
          valueFrom: {configMapKeyRef: {name: app-konfig, key: farbe}}
        volumeMounts:
        - {name: konfig, mountPath: /etc/app/farbe, subPath: farbe}
        - {name: konfig, mountPath: /konfig}
      volumes:
      - name: konfig
        configMap: {name: app-konfig}
EOF
kubectl apply -f konfig.yaml
kubectl -n tag17 rollout status deploy/leser
kubectl -n tag17 patch configmap app-konfig -p '{"data":{"farbe":"rot"}}'

The key farbe (‘color’) changes from blau (‘blue’) to rot (‘red’); leser means ‘reader’.

Symptom (after about a minute):

# on cp1
sleep 60
kubectl -n tag17 exec deploy/leser -- \
  sh -c 'echo env=$FARBE subPath=$(cat /etc/app/farbe) volume=$(cat /konfig/farbe)'

Expected:

env=blau subPath=blau volume=rot

Cause: Environment variables are only set at startup. A subPath mount is a single file fixed at startup – the kubelet never replaces it. Only the volume without subPath follows the ConfigMap (via a symlink, after up to about a minute).

Fix: Roll out again – or have the application read from the volume directory:

# on cp1
kubectl -n tag17 rollout restart deploy/leser
kubectl -n tag17 rollout status deploy/leser

Verify:

# on cp1
kubectl -n tag17 exec deploy/leser -- \
  sh -c 'echo env=$FARBE subPath=$(cat /etc/app/farbe) volume=$(cat /konfig/farbe)'

Expected:

env=rot subPath=rot volume=rot

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the conditions of the Deployment web with their reason. Target time: 30 s
  2. Show all Helm releases in the cluster that are not deployed. Target time: 20 s
  3. Show all failed Jobs in the cluster. Target time: 30 s

Solution

# on cp1
J='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl -n tag17 get deploy web -o jsonpath="$J"
helm list -A --pending --failed -q; echo "rc=$?"
kubectl get jobs -A | grep -E 'NAME|Failed' | tr -s ' '

Expected:

Available=True MinimumReplicasAvailable
Progressing=True NewReplicaSetAvailable
rc=0
NAMESPACE NAME STATUS COMPLETIONS DURATION AGE
tag17 import Failed 0/1 89s 89s

-q only prints names – none here, the Helm release is clean after Drill 3.

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag17 get deploy
helm -n tag17 status demo -o json | jq -r .info.status

Expected:

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
demo    1/1     1            1           3m25s
leser   1/1     1            1           84s
web     2/2     2            2           6m22s
deployed

web is now at 2 – the HPA’s stabilization window has expired.

Challenge

The Deployment api gets a new readiness probe. For two minutes, the rollout has reported nothing new, there are no errors, and the service is running. Find out what the rollout is stuck on and bring it into a clean state.

# on cp1
cat > api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api, namespace: tag17}
spec:
  replicas: 3
  strategy:
    rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
  selector: {matchLabels: {app: api}}
  template:
    metadata: {labels: {app: api}}
    spec:
      containers:
      - name: api
        image: traefik/whoami:v1.11
        readinessProbe:
          httpGet: {path: /, port: 80}
EOF
kubectl apply -f api.yaml
kubectl -n tag17 rollout status deploy/api
P=/spec/template/spec/containers/0/readinessProbe/httpGet/port
kubectl -n tag17 patch deploy api --type=json \
  -p '[{"op": "replace", "path": "'$P'", "value": 8080}]'

Hint: With the default deadline of 600 seconds, ProgressDeadlineExceeded only shows up after ten minutes – look at the ReplicaSets.

Solution

# on cp1
sleep 40
kubectl -n tag17 rollout status deploy/api --timeout=5s
kubectl -n tag17 get rs -l app=api
kubectl -n tag17 get pods -l app=api
kubectl -n tag17 events --types=Warning | grep 'Pod/api' | grep -m1 Unhealthy \
  | cut -c1-200 | fold -w 86

Expected:

Waiting for deployment "api" rollout to finish: 1 out of 3 new replicas have been upda
ted...
error: timed out waiting for the condition
NAME             DESIRED   CURRENT   READY   AGE
api-6745d6c768   1         1         0       48s
api-77d7d744d8   3         3         3       51s
NAME                   READY   STATUS    RESTARTS   AGE
api-6745d6c768-dtp5c   0/1     Running   0          49s
api-77d7d744d8-c2f8c   1/1     Running   0          52s
api-77d7d744d8-hxdzb   1/1     Running   0          52s
api-77d7d744d8-rqq4m   1/1     Running   0          52s
7s (x7 over 47s)        Warning   Unhealthy                      Pod/api-6745d6c768-dt
p5c      Readiness probe failed: Get "http://10.42.0.168:8080/": dial tcp 10.42.0.168:
8080: connect: connection re

The new Pod is running but never becomes ready – the probe asks port 8080, whoami listens on 80. With maxUnavailable: 0, all three old Pods stay: no outage, but no progress either. Go back to the last good version:

# on cp1
kubectl -n tag17 rollout undo deploy/api 2>&1 | tail -1
kubectl -n tag17 rollout status deploy/api
kubectl -n tag17 get rs -l app=api

Expected (abridged):

deployment.apps/api rolled back
deployment "api" successfully rolled out
NAME             DESIRED   CURRENT   READY   AGE
api-6745d6c768   0         0         0       53s
api-77d7d744d8   3         3         3       56s

Save your progress? Sign up for free

Cleanup

# on cp1
helm uninstall demo -n tag17
kubectl delete namespace tag17
helm list -A | grep -c demo

Expected:

release "demo" uninstalled
namespace "tag17" deleted
0

Interview Note

  • Rollout stuck – how do I see it? rollout status aborts with exceeded its progress deadline; get rs shows the new ReplicaSet without ready Pods. The controller does not roll back by itself.
  • HPA shows <unknown>? Events on the HPA: missing request or missing metrics API (kubectl get apiservice, Day 16).
  • Helm another operation is in progress? helm history shows pending-*; helm rollback to the last good revision, never delete the release Secrets by hand.

Enterprise Note

  • Choose progressDeadlineSeconds short enough that CI pipelines fail with rollout status instead of waiting an hour; automatic rollback belongs in the pipeline or in Argo Rollouts/Flagger.
  • Always run Helm in CI with --rollback-on-failure (in Helm 3: --atomic) and a timeout; prevent parallel pipelines on the same release.
  • Enforce requests for all containers (LimitRange, Kyverno), otherwise every HPA is blind.
  • Deliver configuration via volumes without subPath or with a hash in the Pod template (checksum annotation in the Helm chart), so that changes trigger a rollout.