Day 3: Pods Stuck in Pending
Goal: You read from the scheduler’s message why a Pod finds no node, and you track down the Pod that does not exist at all. · Duration: approx. 60 min · Prerequisite: Day 1
Briefing
Pendingmeans: the Pod exists but is not assigned to a node. The scheduler writes the reason as aFailedSchedulingevent on the Pod – with a reason for every node.- The message counts nodes per reason:
0/3 nodes are available: 3 Insufficient cpu. The reasons add up to the number of nodes. - If there is no Pod at all, an admission check rejected it. The event is then on the ReplicaSet.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Requests too large | Insufficient cpu |
Event on the Pod, describe node |
| nodeSelector, taint | untolerated taint(s) |
Event, labels, taints |
| PVC without StorageClass | unbound immediate PVCs |
Event on the PVC |
| Wrong schedulerName | Pending, no events |
.spec.schedulerName |
| ResourceQuota | Deployment 0/2, no Pods |
Event on the ReplicaSet |
Drill 1: Requests too large
Break it:
# on cp1
mkdir -p ~/debug/tag03 && cd ~/debug/tag03
kubectl create namespace tag03
cat > gross.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: gross, namespace: tag03}
spec:
replicas: 1
selector: {matchLabels: {app: gross}}
template:
metadata: {labels: {app: gross}}
spec:
containers:
- name: app
image: busybox:1.37
command: [sleep, infinity]
resources:
requests: {cpu: "6", memory: 256Mi}
limits: {memory: 256Mi}
EOF
kubectl apply -f gross.yamlSymptom:
# on cp1
kubectl -n tag03 get podsExpected:
NAME READY STATUS RESTARTS AGE
gross-67c948f4bc-f8r7z 0/1 Pending 0 5s
Investigate:
# on cp1
kubectl -n tag03 describe pod -l app=gross | sed -n '/^Events/,$p'
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory
kubectl describe node worker1 | sed -n '/Allocated resources/,/memory/p'Expected (line wrapped):
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5s default-scheduler 0/3 nodes are available:
3 Insufficient cpu. preemption: 0/3 nodes are available: 3 Preemption is not
helpful for scheduling.
NAME CPU MEM
cp1 4 12248236Ki
worker1 2 4009840Ki
worker2 2 4009840Ki
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 600m (30%) 100m (5%)
memory 256Mi (6%) 128Mi (3%)
Cause: No node has 6 CPUs to hand out; the largest
(cp1) has 4. The scheduler works with requests, not with
actual load: on worker1, system Pods already reserve 600m,
leaving 1400m free.
Fix:
# on cp1
kubectl -n tag03 set resources deploy/gross --requests=cpu=500m
kubectl -n tag03 rollout status deploy/grossVerify:
# on cp1
kubectl -n tag03 get pods -l app=gross -o wide | cut -c1-80Expected:
NAME READY STATUS RESTARTS AGE IP NODE NOMINA
gross-6879f5c758-fqfbj 1/1 Running 0 2s 10.42.0.19 cp1 <none>
Why? preemption: … not helpful means that even evicting
lower-priority Pods would not be enough. Otherwise, with
PriorityClasses, the scheduler could make room.
Save your progress? Sign up for free
Drill 2: nodeSelector and taint
worker2 becomes a batch node: a label for selection, a
taint against all other Pods.
Break it:
# on cp1
kubectl label node worker2 pool=batch
kubectl taint node worker2 pool=batch:NoSchedule
cat > batch.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: batch, namespace: tag03}
spec:
replicas: 1
selector: {matchLabels: {app: batch}}
template:
metadata: {labels: {app: batch}}
spec:
nodeSelector: {pool: batch}
containers:
- name: app
image: busybox:1.37
command: [sleep, infinity]
resources:
requests: {cpu: 50m, memory: 32Mi}
EOF
kubectl apply -f batch.yamlSymptom:
# on cp1
kubectl -n tag03 get pods -l app=batch
kubectl -n tag03 events --for $(kubectl -n tag03 get pod -l app=batch -o name) \
| tail -1Expected (line wrapped):
NAME READY STATUS RESTARTS AGE
batch-78ccfd557-7rrn6 0/1 Pending 0 4s
4s Warning FailedScheduling Pod/batch-78ccfd557-7rrn6 0/3 nodes are
available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node
affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not
helpful for scheduling.
1 + 2 = 3: one node matches the selector but has the taint; the other two do not match the selector.
Investigate:
# on cp1
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
POOL:.metadata.labels.pool,TAINTS:.spec.taints
kubectl -n tag03 get pod -l app=batch \
-o jsonpath='{range .items[0].spec.tolerations[*]}{.key}{"\n"}{end}'Expected:
NAME POOL TAINTS
cp1 <none> <none>
worker1 <none> <none>
worker2 batch [map[effect:NoSchedule key:pool value:batch]]
node.kubernetes.io/not-ready
node.kubernetes.io/unreachable
Cause: The Pod only has the two tolerations every
Pod gets automatically (300 s for not-ready and
unreachable, Day 14). One for pool=batch is
missing.
Fix:
# on cp1
kubectl -n tag03 patch deploy batch --type=merge -p '
spec:
template:
spec:
tolerations: [{key: pool, value: batch, effect: NoSchedule}]'
kubectl -n tag03 rollout status deploy/batchVerify:
# on cp1
kubectl -n tag03 get pods -l app=batch -o wide | cut -c1-80Expected:
NAME READY STATUS RESTARTS AGE IP NODE NOMINA
batch-84487547f6-wff6g 1/1 Running 0 2s 10.42.2.8 worker2 <none>
Why? A toleration only allows the tainted node; it
does not pull the Pod there. Only the nodeSelector makes
sure it lands on worker2.
Save your progress? Sign up for free
Drill 3: PVC without StorageClass
Break it:
# on cp1
cat > datenbank.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: daten, namespace: tag03}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: datenbank, namespace: tag03}
spec:
replicas: 1
selector: {matchLabels: {app: datenbank}}
template:
metadata: {labels: {app: datenbank}}
spec:
containers:
- name: app
image: busybox:1.37
command: [sh, -c, 'date >> /daten/start.log; sleep infinity']
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: daten}
EOF
kubectl apply -f datenbank.yamlSymptom:
# on cp1
kubectl -n tag03 events --for $(kubectl -n tag03 get pod -l app=datenbank -o name) \
| tail -1Expected (line wrapped):
6s Warning FailedScheduling Pod/datenbank-595dffc769-fpjlx 0/3 nodes are
available: pod has unbound immediate PersistentVolumeClaims. not found
Investigate:
# on cp1
kubectl -n tag03 get pvc daten | cut -c1-72
kubectl -n tag03 describe pvc daten | sed -n '/^Events/,$p'Expected (line wrapped):
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUT
daten Pending <unset>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 1s (x2 over 7s) persistentvolume-controller no persistent
volumes available for this claim and no storage class is set
Cause: The PVC names no StorageClass, and the cluster has no default StorageClass (Day 0, on purpose). Without a class, Kubernetes looks for a matching existing PV – and there is none. The message on the Pod is misleadingly terse; the real answer is on the PVC.
Fix:
# on cp1
kubectl -n tag03 patch pvc daten -p '{"spec":{"storageClassName":"local-path"}}'Verify:
# on cp1
kubectl -n tag03 get pvc daten -o custom-columns=NAME:.metadata.name,\
STATUS:.status.phase,SC:.spec.storageClassName
kubectl -n tag03 get pods -l app=datenbank -o wide | cut -c1-80Expected:
NAME STATUS SC
daten Bound local-path
NAME READY STATUS RESTARTS AGE IP NODE
datenbank-595dffc769-fpjlx 1/1 Running 0 35s 10.42.1.14 worker1
Why? A missing storageClassName may be set afterwards
(but not an explicitly empty value ""). A class that has
been set, however, cannot be changed:
# on cp1
kubectl -n tag03 patch pvc daten -p '{"spec":{"storageClassName":"schnell"}}' \
2>&1 | head -2Expected (line wrapped):
The PersistentVolumeClaim "daten" is invalid: spec: Forbidden: spec is immutable
after creation except resources.requests and volumeAttributesClassName for bound
claims
Then the only option is to delete the PVC and create it again – along with any data already in it.
Save your progress? Sign up for free
Drill 4: Pending without a single event
Break it:
# on cp1
cat > still.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: still, namespace: tag03}
spec:
schedulerName: batch-scheduler
containers:
- name: app
image: busybox:1.37
command: [sleep, infinity]
EOF
kubectl apply -f still.yamlSymptom:
# on cp1
kubectl -n tag03 get pod still -o wide | cut -c1-72
kubectl -n tag03 events --for pod/stillExpected:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
still 0/1 Pending 0 15s <none> <none> <none>
No events found in tag03 namespace.
Investigate: No FailedScheduling means
that no scheduler has even looked at the Pod.
# on cp1
kubectl -n tag03 get pod still -o jsonpath='{.spec.schedulerName}{"\n"}'Expected:
batch-scheduler
Cause: The Pod asks for a scheduler
batch-scheduler that does not exist in the cluster. The
default scheduler ignores it, and nobody reports anything.
Fix: schedulerName is immutable after
creation:
# on cp1
kubectl -n tag03 patch pod still -p '{"spec":{"schedulerName":"default-scheduler"}}' \
2>&1 | head -2
kubectl -n tag03 delete pod still
sed -i 's/batch-scheduler/default-scheduler/' still.yaml
kubectl apply -f still.yamlExpected (line wrapped):
The Pod "still" is invalid: spec: Forbidden: pod updates may not change fields other
than `spec.containers[*].image`,`spec.initContainers[*].image`, ...
pod "still" deleted from tag03 namespace
pod/still created
Verify:
# on cp1
kubectl -n tag03 wait pod/still --for=condition=Ready
kubectl -n tag03 events --for pod/still | head -2 | cut -c1-80Expected:
pod/still condition met
LAST SEEN TYPE REASON OBJECT MESSAGE
2s Normal Scheduled Pod/still Successfully assigned tag03/still to worker
A cluster whose scheduler is down shows the same picture –
Pending without events (Day 15).
The Pod has done its job. Delete it before you create the quota in Drill 5 – otherwise the quota counts it too:
# on cp1
kubectl -n tag03 delete pod stillExpected:
pod "still" deleted from tag03 namespace
Save your progress? Sign up for free
Drill 5: ResourceQuota – the Pod that does not exist
Break it:
# on cp1
cat > quota.yaml <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata: {name: grenze, namespace: tag03}
spec:
hard: {pods: "10", requests.cpu: "2", requests.memory: 2Gi, limits.memory: 4Gi}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag03}
spec:
replicas: 2
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
containers:
- name: web
image: nginxinc/nginx-unprivileged:1.29
ports: [{containerPort: 8080}]
EOF
kubectl apply -f quota.yamlSymptom:
# on cp1
kubectl -n tag03 get deploy web
kubectl -n tag03 get pods -l app=webExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
web 0/2 0 0 6s
No resources found in tag03 namespace.
No Pod, so no describe pod either. Ask one level up:
Investigate:
# on cp1
kubectl -n tag03 describe deploy web | sed -n '/^Conditions/,/^NewReplicaSet/p'
kubectl -n tag03 events --types=Warning | grep ReplicaSet/web | tail -1Expected (lines wrapped):
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetCreated
Available False MinimumReplicasUnavailable
ReplicaFailure True FailedCreate
OldReplicaSets: <none>
NewReplicaSet: web-f48975487 (0/2 replicas created)
1s (x2 over 4s) Warning FailedCreate ReplicaSet/web-f48975487 (combined from
similar events): Error creating: pods "web-f48975487-c9hdr" is forbidden: failed
quota: grenze: must specify limits.memory for: web; requests.cpu for: web;
requests.memory for: web
Cause: If a quota limits requests.cpu
or limits.memory, every new Pod must specify those values.
The container web has no resources at all, so
admission rejects every Pod. The ReplicaSet controller keeps retrying,
with backoff.
Fix:
# on cp1
kubectl -n tag03 set resources deploy/web --requests=cpu=100m,memory=64Mi \
--limits=memory=128Mi
kubectl -n tag03 rollout status deploy/webVerify:
# on cp1
kubectl -n tag03 get deploy web
kubectl -n tag03 describe resourcequota grenze | tail -5Expected:
NAME READY UP-TO-DATE AVAILABLE AGE
web 2/2 2 2 48s
limits.memory 512Mi 4Gi
pods 5 10
requests.cpu 750m 2
requests.memory 416Mi 2Gi
The quota counts all Pods in the namespace, including
gross, batch and datenbank.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all Pods in the cluster that are
Pending. Target time: 20 s - Show the taints of all nodes. Target time: 20 s
- Show how much of the quota in
tag03is used. Target time: 10 s - Show how much CPU is already reserved on
worker1. Target time: 20 s
Solution
# on cp1
kubectl get pods -A --field-selector=status.phase=Pending
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
kubectl -n tag03 get resourcequota
kubectl describe node worker1 | grep -A4 'Allocated resources'Expected (abridged):
No resources found
NAME TAINTS
cp1 <none>
worker1 <none>
worker2 [map[effect:NoSchedule key:pool value:batch]]
NAME REQUEST LIMIT ...
grenze pods: 5/10, requests.cpu: 750m/2, requests.memory: 416Mi/2Gi limits.m ...
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 600m (30%) 100m (5%)
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag03 get deployExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
batch 1/1 1 1 2m40s
datenbank 1/1 1 1 2m19s
gross 1/1 1 1 3m2s
web 2/2 2 2 92s
# on cp1
kubectl get pods -A --field-selector=status.phase=PendingExpected:
No resources found
Challenge
In tag03, create a LimitRange that allows at most 500m
CPU and 256Mi per container (default limit 200m/128Mi, default request
100m/64Mi), and a Deployment rechner with
limits: {cpu: "1", memory: 128Mi}. Find out why no Pod
appears, fix it and check which requests the Pod ends up with.
Hint: Here, too, the message is on the ReplicaSet.
Solution
# on cp1
cat > rechner.yaml <<'EOF'
apiVersion: v1
kind: LimitRange
metadata: {name: grenzen, namespace: tag03}
spec:
limits:
- type: Container
max: {cpu: 500m, memory: 256Mi}
default: {cpu: 200m, memory: 128Mi}
defaultRequest: {cpu: 100m, memory: 64Mi}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: rechner, namespace: tag03}
spec:
replicas: 1
selector: {matchLabels: {app: rechner}}
template:
metadata: {labels: {app: rechner}}
spec:
containers:
- name: app
image: busybox:1.37
command: [sleep, infinity]
resources:
limits: {cpu: "1", memory: 128Mi}
EOF
kubectl apply -f rechner.yaml
kubectl -n tag03 events --types=Warning | grep ReplicaSet/rechner | tail -1
kubectl -n tag03 set resources deploy/rechner --limits=cpu=500m,memory=128Mi
kubectl -n tag03 rollout status deploy/rechner
kubectl -n tag03 get pod -l app=rechner \
-o jsonpath='{.items[0].spec.containers[0].resources}{"\n"}'Expected (line wrapped):
2s Warning FailedCreate ReplicaSet/rechner-5dc45f7db6 (combined from
similar events): Error creating: pods "rechner-5dc45f7db6-j9mmr" is forbidden:
maximum cpu usage per Container is 500m, but limit is 1
deployment.apps/rechner resource requirements updated
deployment "rechner" successfully rolled out
{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"500m","memory":"128Mi"}}
Why a 500m request and not the default of 100m? If the manifest
contains only a limit, Kubernetes sets the request equal to the limit
during defaulting; the LimitRange’s defaultRequest only
applies when there is no limit either.
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag03
kubectl taint node worker2 pool=batch:NoSchedule-
kubectl label node worker2 pool-
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
POOL:.metadata.labels.pool,TAINTS:.spec.taints
kubectl get pvExpected:
namespace "tag03" deleted
node/worker2 untainted
node/worker2 unlabeled
NAME POOL TAINTS
cp1 <none> <none>
worker1 <none> <none>
worker2 <none> <none>
No resources found
Interview Note
- A Pod is stuck in
Pending– where do you look? Events on the Pod (FailedScheduling); the counts per reason add up to the number of nodes. For volumes, move on to the PVC. - The Deployment shows
0/2, but there are no Pods? Admission rejected them (quota, LimitRange, Pod Security, webhook) – event on the ReplicaSet or conditionReplicaFailureon the Deployment. - Taint or nodeSelector? A taint keeps foreign Pods away, nodeSelector/affinity pulls your own Pods in; for dedicated nodes you need both.
Enterprise Note
- Quotas per team namespace are common; a LimitRange with defaults
next to them prevents Pods without
resourcesfrom being rejected. - The Cluster Autoscaler only adds nodes if a new node from a node group could take the Pod. If the toleration, a matching label or the StorageClass is missing, the Pod waits forever.
- Define exactly one default StorageClass per cluster and document it; two defaults are a fault pattern too.
- PriorityClasses decide who gets evicted when resources run short – platform Pods higher than applications.