Skip to content
Hands-on IT IT Drills

Day 3: Pods Stuck in Pending

Goal: You read from the scheduler’s message why a Pod finds no node, and you track down the Pod that does not exist at all. · Duration: approx. 60 min · Prerequisite: Day 1

Briefing

  • Pending means: the Pod exists but is not assigned to a node. The scheduler writes the reason as a FailedScheduling event on the Pod – with a reason for every node.
  • The message counts nodes per reason: 0/3 nodes are available: 3 Insufficient cpu. The reasons add up to the number of nodes.
  • If there is no Pod at all, an admission check rejected it. The event is then on the ReplicaSet.
Fault Symptom Where the answer is
Requests too large Insufficient cpu Event on the Pod, describe node
nodeSelector, taint untolerated taint(s) Event, labels, taints
PVC without StorageClass unbound immediate PVCs Event on the PVC
Wrong schedulerName Pending, no events .spec.schedulerName
ResourceQuota Deployment 0/2, no Pods Event on the ReplicaSet

Drill 1: Requests too large

Break it:

# on cp1
mkdir -p ~/debug/tag03 && cd ~/debug/tag03
kubectl create namespace tag03
cat > gross.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: gross, namespace: tag03}
spec:
  replicas: 1
  selector: {matchLabels: {app: gross}}
  template:
    metadata: {labels: {app: gross}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command: [sleep, infinity]
        resources:
          requests: {cpu: "6", memory: 256Mi}
          limits: {memory: 256Mi}
EOF
kubectl apply -f gross.yaml

Symptom:

# on cp1
kubectl -n tag03 get pods

Expected:

NAME                     READY   STATUS    RESTARTS   AGE
gross-67c948f4bc-f8r7z   0/1     Pending   0          5s

Investigate:

# on cp1
kubectl -n tag03 describe pod -l app=gross | sed -n '/^Events/,$p'
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory
kubectl describe node worker1 | sed -n '/Allocated resources/,/memory/p'

Expected (line wrapped):

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  5s    default-scheduler  0/3 nodes are available:
  3 Insufficient cpu. preemption: 0/3 nodes are available: 3 Preemption is not
  helpful for scheduling.
NAME      CPU   MEM
cp1       4     12248236Ki
worker1   2     4009840Ki
worker2   2     4009840Ki
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                600m (30%)  100m (5%)
  memory             256Mi (6%)  128Mi (3%)

Cause: No node has 6 CPUs to hand out; the largest (cp1) has 4. The scheduler works with requests, not with actual load: on worker1, system Pods already reserve 600m, leaving 1400m free.

Fix:

# on cp1
kubectl -n tag03 set resources deploy/gross --requests=cpu=500m
kubectl -n tag03 rollout status deploy/gross

Verify:

# on cp1
kubectl -n tag03 get pods -l app=gross -o wide | cut -c1-80

Expected:

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINA
gross-6879f5c758-fqfbj   1/1     Running   0          2s    10.42.0.19   cp1    <none>

Why? preemption: … not helpful means that even evicting lower-priority Pods would not be enough. Otherwise, with PriorityClasses, the scheduler could make room.

Save your progress? Sign up for free

Drill 2: nodeSelector and taint

worker2 becomes a batch node: a label for selection, a taint against all other Pods.

Break it:

# on cp1
kubectl label node worker2 pool=batch
kubectl taint node worker2 pool=batch:NoSchedule
cat > batch.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: batch, namespace: tag03}
spec:
  replicas: 1
  selector: {matchLabels: {app: batch}}
  template:
    metadata: {labels: {app: batch}}
    spec:
      nodeSelector: {pool: batch}
      containers:
      - name: app
        image: busybox:1.37
        command: [sleep, infinity]
        resources:
          requests: {cpu: 50m, memory: 32Mi}
EOF
kubectl apply -f batch.yaml

Symptom:

# on cp1
kubectl -n tag03 get pods -l app=batch
kubectl -n tag03 events --for $(kubectl -n tag03 get pod -l app=batch -o name) \
  | tail -1

Expected (line wrapped):

NAME                    READY   STATUS    RESTARTS   AGE
batch-78ccfd557-7rrn6   0/1     Pending   0          4s
4s          Warning   FailedScheduling   Pod/batch-78ccfd557-7rrn6   0/3 nodes are
  available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node
  affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not
  helpful for scheduling.

1 + 2 = 3: one node matches the selector but has the taint; the other two do not match the selector.

Investigate:

# on cp1
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
POOL:.metadata.labels.pool,TAINTS:.spec.taints
kubectl -n tag03 get pod -l app=batch \
  -o jsonpath='{range .items[0].spec.tolerations[*]}{.key}{"\n"}{end}'

Expected:

NAME      POOL     TAINTS
cp1       <none>   <none>
worker1   <none>   <none>
worker2   batch    [map[effect:NoSchedule key:pool value:batch]]
node.kubernetes.io/not-ready
node.kubernetes.io/unreachable

Cause: The Pod only has the two tolerations every Pod gets automatically (300 s for not-ready and unreachable, Day 14). One for pool=batch is missing.

Fix:

# on cp1
kubectl -n tag03 patch deploy batch --type=merge -p '
spec:
  template:
    spec:
      tolerations: [{key: pool, value: batch, effect: NoSchedule}]'
kubectl -n tag03 rollout status deploy/batch

Verify:

# on cp1
kubectl -n tag03 get pods -l app=batch -o wide | cut -c1-80

Expected:

NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE      NOMINA
batch-84487547f6-wff6g   1/1     Running   0          2s    10.42.2.8   worker2   <none>

Why? A toleration only allows the tainted node; it does not pull the Pod there. Only the nodeSelector makes sure it lands on worker2.

Save your progress? Sign up for free

Drill 3: PVC without StorageClass

Break it:

# on cp1
cat > datenbank.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: daten, namespace: tag03}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: datenbank, namespace: tag03}
spec:
  replicas: 1
  selector: {matchLabels: {app: datenbank}}
  template:
    metadata: {labels: {app: datenbank}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command: [sh, -c, 'date >> /daten/start.log; sleep infinity']
        volumeMounts: [{name: daten, mountPath: /daten}]
      volumes:
      - name: daten
        persistentVolumeClaim: {claimName: daten}
EOF
kubectl apply -f datenbank.yaml

Symptom:

# on cp1
kubectl -n tag03 events --for $(kubectl -n tag03 get pod -l app=datenbank -o name) \
  | tail -1

Expected (line wrapped):

6s          Warning   FailedScheduling   Pod/datenbank-595dffc769-fpjlx   0/3 nodes are
  available: pod has unbound immediate PersistentVolumeClaims. not found

Investigate:

# on cp1
kubectl -n tag03 get pvc daten | cut -c1-72
kubectl -n tag03 describe pvc daten | sed -n '/^Events/,$p'

Expected (line wrapped):

NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUT
daten   Pending                                                     <unset>
Events:
  Type    Reason         Age              From                         Message
  ----    ------         ----             ----                         -------
  Normal  FailedBinding  1s (x2 over 7s)  persistentvolume-controller  no persistent
  volumes available for this claim and no storage class is set

Cause: The PVC names no StorageClass, and the cluster has no default StorageClass (Day 0, on purpose). Without a class, Kubernetes looks for a matching existing PV – and there is none. The message on the Pod is misleadingly terse; the real answer is on the PVC.

Fix:

# on cp1
kubectl -n tag03 patch pvc daten -p '{"spec":{"storageClassName":"local-path"}}'

Verify:

# on cp1
kubectl -n tag03 get pvc daten -o custom-columns=NAME:.metadata.name,\
STATUS:.status.phase,SC:.spec.storageClassName
kubectl -n tag03 get pods -l app=datenbank -o wide | cut -c1-80

Expected:

NAME    STATUS   SC
daten   Bound    local-path
NAME                         READY   STATUS    RESTARTS   AGE   IP           NODE
datenbank-595dffc769-fpjlx   1/1     Running   0          35s   10.42.1.14   worker1

Why? A missing storageClassName may be set afterwards (but not an explicitly empty value ""). A class that has been set, however, cannot be changed:

# on cp1
kubectl -n tag03 patch pvc daten -p '{"spec":{"storageClassName":"schnell"}}' \
  2>&1 | head -2

Expected (line wrapped):

The PersistentVolumeClaim "daten" is invalid: spec: Forbidden: spec is immutable
  after creation except resources.requests and volumeAttributesClassName for bound
  claims

Then the only option is to delete the PVC and create it again – along with any data already in it.

Save your progress? Sign up for free

Drill 4: Pending without a single event

Break it:

# on cp1
cat > still.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: still, namespace: tag03}
spec:
  schedulerName: batch-scheduler
  containers:
  - name: app
    image: busybox:1.37
    command: [sleep, infinity]
EOF
kubectl apply -f still.yaml

Symptom:

# on cp1
kubectl -n tag03 get pod still -o wide | cut -c1-72
kubectl -n tag03 events --for pod/still

Expected:

NAME    READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE
still   0/1     Pending   0          15s   <none>   <none>   <none>
No events found in tag03 namespace.

Investigate: No FailedScheduling means that no scheduler has even looked at the Pod.

# on cp1
kubectl -n tag03 get pod still -o jsonpath='{.spec.schedulerName}{"\n"}'

Expected:

batch-scheduler

Cause: The Pod asks for a scheduler batch-scheduler that does not exist in the cluster. The default scheduler ignores it, and nobody reports anything.

Fix: schedulerName is immutable after creation:

# on cp1
kubectl -n tag03 patch pod still -p '{"spec":{"schedulerName":"default-scheduler"}}' \
  2>&1 | head -2
kubectl -n tag03 delete pod still
sed -i 's/batch-scheduler/default-scheduler/' still.yaml
kubectl apply -f still.yaml

Expected (line wrapped):

The Pod "still" is invalid: spec: Forbidden: pod updates may not change fields other
  than `spec.containers[*].image`,`spec.initContainers[*].image`, ...
pod "still" deleted from tag03 namespace
pod/still created

Verify:

# on cp1
kubectl -n tag03 wait pod/still --for=condition=Ready
kubectl -n tag03 events --for pod/still | head -2 | cut -c1-80

Expected:

pod/still condition met
LAST SEEN   TYPE     REASON      OBJECT      MESSAGE
2s          Normal   Scheduled   Pod/still   Successfully assigned tag03/still to worker

A cluster whose scheduler is down shows the same picture – Pending without events (Day 15).

The Pod has done its job. Delete it before you create the quota in Drill 5 – otherwise the quota counts it too:

# on cp1
kubectl -n tag03 delete pod still

Expected:

pod "still" deleted from tag03 namespace

Save your progress? Sign up for free

Drill 5: ResourceQuota – the Pod that does not exist

Break it:

# on cp1
cat > quota.yaml <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata: {name: grenze, namespace: tag03}
spec:
  hard: {pods: "10", requests.cpu: "2", requests.memory: 2Gi, limits.memory: 4Gi}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag03}
spec:
  replicas: 2
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: nginxinc/nginx-unprivileged:1.29
        ports: [{containerPort: 8080}]
EOF
kubectl apply -f quota.yaml

Symptom:

# on cp1
kubectl -n tag03 get deploy web
kubectl -n tag03 get pods -l app=web

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    0/2     0            0           6s
No resources found in tag03 namespace.

No Pod, so no describe pod either. Ask one level up:

Investigate:

# on cp1
kubectl -n tag03 describe deploy web | sed -n '/^Conditions/,/^NewReplicaSet/p'
kubectl -n tag03 events --types=Warning | grep ReplicaSet/web | tail -1

Expected (lines wrapped):

Conditions:
  Type             Status  Reason
  ----             ------  ------
  Progressing      True    NewReplicaSetCreated
  Available        False   MinimumReplicasUnavailable
  ReplicaFailure   True    FailedCreate
OldReplicaSets:    <none>
NewReplicaSet:     web-f48975487 (0/2 replicas created)
1s (x2 over 4s)   Warning   FailedCreate   ReplicaSet/web-f48975487   (combined from
  similar events): Error creating: pods "web-f48975487-c9hdr" is forbidden: failed
  quota: grenze: must specify limits.memory for: web; requests.cpu for: web;
  requests.memory for: web

Cause: If a quota limits requests.cpu or limits.memory, every new Pod must specify those values. The container web has no resources at all, so admission rejects every Pod. The ReplicaSet controller keeps retrying, with backoff.

Fix:

# on cp1
kubectl -n tag03 set resources deploy/web --requests=cpu=100m,memory=64Mi \
  --limits=memory=128Mi
kubectl -n tag03 rollout status deploy/web

Verify:

# on cp1
kubectl -n tag03 get deploy web
kubectl -n tag03 describe resourcequota grenze | tail -5

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    2/2     2            2           48s
limits.memory    512Mi  4Gi
pods             5      10
requests.cpu     750m   2
requests.memory  416Mi  2Gi

The quota counts all Pods in the namespace, including gross, batch and datenbank.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all Pods in the cluster that are Pending. Target time: 20 s
  2. Show the taints of all nodes. Target time: 20 s
  3. Show how much of the quota in tag03 is used. Target time: 10 s
  4. Show how much CPU is already reserved on worker1. Target time: 20 s

Solution

# on cp1
kubectl get pods -A --field-selector=status.phase=Pending
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
kubectl -n tag03 get resourcequota
kubectl describe node worker1 | grep -A4 'Allocated resources'

Expected (abridged):

No resources found
NAME      TAINTS
cp1       <none>
worker1   <none>
worker2   [map[effect:NoSchedule key:pool value:batch]]
NAME     REQUEST                                                        LIMIT    ...
grenze   pods: 5/10, requests.cpu: 750m/2, requests.memory: 416Mi/2Gi   limits.m ...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                600m (30%)  100m (5%)

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag03 get deploy

Expected:

NAME        READY   UP-TO-DATE   AVAILABLE   AGE
batch       1/1     1            1           2m40s
datenbank   1/1     1            1           2m19s
gross       1/1     1            1           3m2s
web         2/2     2            2           92s
# on cp1
kubectl get pods -A --field-selector=status.phase=Pending

Expected:

No resources found

Challenge

In tag03, create a LimitRange that allows at most 500m CPU and 256Mi per container (default limit 200m/128Mi, default request 100m/64Mi), and a Deployment rechner with limits: {cpu: "1", memory: 128Mi}. Find out why no Pod appears, fix it and check which requests the Pod ends up with.

Hint: Here, too, the message is on the ReplicaSet.

Solution

# on cp1
cat > rechner.yaml <<'EOF'
apiVersion: v1
kind: LimitRange
metadata: {name: grenzen, namespace: tag03}
spec:
  limits:
  - type: Container
    max: {cpu: 500m, memory: 256Mi}
    default: {cpu: 200m, memory: 128Mi}
    defaultRequest: {cpu: 100m, memory: 64Mi}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: rechner, namespace: tag03}
spec:
  replicas: 1
  selector: {matchLabels: {app: rechner}}
  template:
    metadata: {labels: {app: rechner}}
    spec:
      containers:
      - name: app
        image: busybox:1.37
        command: [sleep, infinity]
        resources:
          limits: {cpu: "1", memory: 128Mi}
EOF
kubectl apply -f rechner.yaml
kubectl -n tag03 events --types=Warning | grep ReplicaSet/rechner | tail -1
kubectl -n tag03 set resources deploy/rechner --limits=cpu=500m,memory=128Mi
kubectl -n tag03 rollout status deploy/rechner
kubectl -n tag03 get pod -l app=rechner \
  -o jsonpath='{.items[0].spec.containers[0].resources}{"\n"}'

Expected (line wrapped):

2s          Warning   FailedCreate   ReplicaSet/rechner-5dc45f7db6   (combined from
  similar events): Error creating: pods "rechner-5dc45f7db6-j9mmr" is forbidden:
  maximum cpu usage per Container is 500m, but limit is 1
deployment.apps/rechner resource requirements updated
deployment "rechner" successfully rolled out
{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"500m","memory":"128Mi"}}

Why a 500m request and not the default of 100m? If the manifest contains only a limit, Kubernetes sets the request equal to the limit during defaulting; the LimitRange’s defaultRequest only applies when there is no limit either.

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag03
kubectl taint node worker2 pool=batch:NoSchedule-
kubectl label node worker2 pool-
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
POOL:.metadata.labels.pool,TAINTS:.spec.taints
kubectl get pv

Expected:

namespace "tag03" deleted
node/worker2 untainted
node/worker2 unlabeled
NAME      POOL     TAINTS
cp1       <none>   <none>
worker1   <none>   <none>
worker2   <none>   <none>
No resources found

Interview Note

  • A Pod is stuck in Pending – where do you look? Events on the Pod (FailedScheduling); the counts per reason add up to the number of nodes. For volumes, move on to the PVC.
  • The Deployment shows 0/2, but there are no Pods? Admission rejected them (quota, LimitRange, Pod Security, webhook) – event on the ReplicaSet or condition ReplicaFailure on the Deployment.
  • Taint or nodeSelector? A taint keeps foreign Pods away, nodeSelector/affinity pulls your own Pods in; for dedicated nodes you need both.

Enterprise Note

  • Quotas per team namespace are common; a LimitRange with defaults next to them prevents Pods without resources from being rejected.
  • The Cluster Autoscaler only adds nodes if a new node from a node group could take the Pod. If the toleration, a matching label or the StorageClass is missing, the Pod waits forever.
  • Define exactly one default StorageClass per cluster and document it; two defaults are a fault pattern too.
  • PriorityClasses decide who gets evicted when resources run short – platform Pods higher than applications.