Day 5: Scheduling
Goal: You control where Pods land and how much they are allowed to consume – and you see at once why a Pod stays
Pending. · Duration: approx. 75 min · Prerequisite: Day 4
Briefing
CKA: “Configure Pod admission and scheduling (limits, node affinity, etc.)” (part of the 15 %).
- Requests decide placement, limits set the upper bound at runtime.
- LimitRange sets default values, ResourceQuota caps a namespace.
- nodeSelector / affinity pull Pods toward nodes, taints repel them, tolerations allow the exception.
- If a Pod stays
Pending, the reason is in the events (FailedScheduling).
mkdir -p ~/cka/tag05 && cd ~/cka/tag05
k create ns tag05
k config set-context --current --namespace=tag05Drill 1: A Pod That Fits Nowhere
How much of worker1 is already allocated?
k describe node worker1 | grep -A6 'Allocated resources'Expected (abridged):
Allocated resources:
Resource Requests Limits
-------- -------- ------
cpu 200m (10%) 0 (0%)
memory 140Mi (3%) 340Mi (8%)
Now a Pod that requests 8 CPUs – generated and given resources in a single pipe:
k run big --image=nginx:1.29 $do \
| k set resources --local -f - --requests=cpu=8 -o yaml | k apply -f -
k get pod big
k get events --field-selector involvedObject.name=big | tail -1Expected (abridged):
NAME READY STATUS RESTARTS AGE
big 0/1 Pending 0 6s
Warning FailedScheduling pod/big 0/3 nodes are available: 1 node(s) had
untolerated taint(s), 2 Insufficient cpu. ...
Read the message carefully: cp1 is ruled out by the
taint, the workers by the CPU.
k delete pod big --nowSave your progress? Sign up for free
Drill 2: LimitRange and ResourceQuota
k create quota tag05-quota \
--hard=pods=4,requests.cpu=1,requests.memory=1Gi,limits.memory=2Gi
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
spec:
limits:
- type: Container
default:
memory: 256Mi
defaultRequest:
cpu: 100m
memory: 128Mi
EOF
k run small --image=nginx:1.29
k get pod small -o jsonpath='{.spec.containers[0].resources}{"\n"}'Expected:
{"limits":{"memory":"256Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}
The Pod had no resources – the LimitRange filled them in. Without it,
the quota would reject the Pod, because with requests.cpu
in the quota, every Pod must have requests.
k describe quota tag05-quota | tail -6
k create deployment many --image=nginx:1.29 --replicas=6
k get deployment manyExpected:
Resource Used Hard
-------- ---- ----
limits.memory 256Mi 2Gi
pods 1 4
requests.cpu 100m 1
requests.memory 128Mi 1Gi
deployment.apps/many created
NAME READY UP-TO-DATE AVAILABLE AGE
many 3/6 3 3 8s
Only three new Pods – together with small, the quota of
4 is full. You find the reason on the ReplicaSet, not on the
Deployment:
k describe rs -l app=many | grep -m1 'exceeded quota'Expected (abridged):
Warning FailedCreate 8s replicaset-controller Error creating:
pods "many-6df65cc865-lgb8s" is forbidden: exceeded quota: tag05-quota,
requested: pods=1, used: pods=4, limited: pods=4
Clean up:
k delete deployment many
k delete pod small
k delete quota tag05-quota
k delete limitrange defaultsSave your progress? Sign up for free
Drill 3: nodeSelector and Node Affinity
k label node worker2 disk=ssd
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
name: fast
spec:
nodeSelector:
disk: ssd
containers:
- name: fast
image: nginx:1.29
---
apiVersion: v1
kind: Pod
metadata:
name: affine
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: disk
operator: NotIn
values: ["ssd"]
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
containers:
- name: affine
image: nginx:1.29
EOF
k get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeNameExpected:
NAME NODE
affine worker1
fast worker2
Expressions within one matchExpressions are combined
with AND, multiple nodeSelectorTerms with OR.
preferredDuringScheduling… would only be a preference, not
a requirement.
k delete pod fast affine --nowSave your progress? Sign up for free
Drill 4: Taints and Tolerations
worker1 goes into maintenance – new Pods should avoid
it:
k taint node worker1 maintenance=true:NoSchedule
k create deployment spread --image=nginx:1.29 --replicas=3
k rollout status deployment spread
k get pods -l app=spread -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeNameExpected:
NAME NODE
spread-85cb54f4b9-fr7nl worker2
spread-85cb54f4b9-lkjq6 worker2
spread-85cb54f4b9-w8wgl worker2
A Pod with a matching toleration may still go there:
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
name: tolerant
spec:
tolerations:
- key: maintenance
operator: Equal
value: "true"
effect: NoSchedule
nodeSelector:
kubernetes.io/hostname: worker1
containers:
- name: tolerant
image: nginx:1.29
EOF
k get pod tolerant -o wideExpected: tolerant runs on
worker1. The toleration allows the node but does not force
it – here the nodeSelector takes care of that.
Remove the taint again (the - at the end):
k taint node worker1 maintenance=true:NoSchedule-
k describe node worker1 | grep TaintsExpected:
node/worker1 untainted
Taints: <none>
Save your progress? Sign up for free
Drill 5: NoExecute Clears the Node
NoSchedule leaves running Pods alone.
NoExecute evicts them:
k taint node worker2 evict=now:NoExecute
k get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeNameExpected (after about 10 s):
NAME NODE
spread-85cb54f4b9-4vczg worker1
spread-85cb54f4b9-6fsfr worker1
spread-85cb54f4b9-nn2dh worker1
tolerant worker1
All three spread Pods have moved to
worker1.
k taint node worker2 evict=now:NoExecute-
k delete deployment spread
k delete pod tolerant --nowSave your progress? Sign up for free
Drill 6: Spreading Pods – and a Trap
cat > spread.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: spread
spec:
replicas: 4
selector:
matchLabels:
app: spread
template:
metadata:
labels:
app: spread
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: spread
containers:
- name: nginx
image: nginx:1.29
EOF
k apply -f spread.yaml
k get pods -l app=spread -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeNameExpected:
NAME NODE
spread-867df78bcb-ssvvs <none>
spread-867df78bcb-b4fl7 <none>
spread-867df78bcb-mhqbk worker1
spread-867df78bcb-vs768 worker2
Two Pods stay Pending. Why?
k get events --field-selector reason=FailedScheduling | tail -1Expected (abridged):
Warning FailedScheduling pod/spread-867df78bcb-ssvvs 0/3 nodes are available:
1 node(s) had untolerated taint(s), 2 node(s) didn't match pod topology spread
constraints. ...
The scheduler counts cp1 as a topology domain with 0
Pods, even though no Pod can ever land there because of the taint. With
0 on cp1, maxSkew: 1 allows only one Pod per
worker. The fix: take taints into account when spreading.
sed -i \
's/^ whenUnsatisfiable: DoNotSchedule/&\n nodeTaintsPolicy: Honor/' \
spread.yaml
grep -A1 whenUnsatisfiable spread.yaml
k delete deployment spread
k apply -f spread.yaml
k rollout status deployment spread
k get pods -l app=spread -o custom-columns=NODE:.spec.nodeName --no-headers \
| sort | uniq -cExpected:
whenUnsatisfiable: DoNotSchedule
nodeTaintsPolicy: Honor
deployment.apps "spread" deleted from tag05 namespace
deployment.apps/spread created
deployment "spread" successfully rolled out
2 worker1
2 worker2
Why delete instead of running apply on the running
Deployment? Topology spread only takes effect at scheduling time. During
a rolling update, the old Pods are still counted, and the distribution
stays lopsided.
Save your progress? Sign up for free
Drill 7: Static Pod and Manual Scheduling
A static Pod runs without the scheduler – the kubelet starts it from a file. On worker1:
cat <<'EOF' | sudo tee /etc/kubernetes/manifests/static-web.yaml
apiVersion: v1
kind: Pod
metadata:
name: static-web
namespace: tag05
spec:
containers:
- name: nginx
image: nginx:1.29
EOFBack on cp1:
k get pods | grep static
k delete pod static-web-worker1
k get pods | grep staticExpected:
static-web-worker1 1/1 Running 0 16s
pod "static-web-worker1" deleted from tag05 namespace
static-web-worker1 0/1 Pending 0 6s
The Pod is back immediately – its name gets the node as a suffix, and it can only be deleted through the file. On worker1:
sudo rm /etc/kubernetes/manifests/static-web.yamlYou can also bypass the scheduler with nodeName:
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
name: manual
spec:
nodeName: worker2
containers:
- name: manual
image: nginx:1.29
EOF
k get pod manual -o wideExpected: manual runs on
worker2. nodeName ignores affinity and even
NoSchedule taints – useful when the scheduler itself is
broken (Day 14).
Save your progress? Sign up for free
Exercise: Speed Drills
- Create a PriorityClass
criticalwith the value 1000000. Target time: 20 s - Label
worker1withzone=a. Target time: 10 s - Make
worker2unusable for new Pods without disturbing running Pods, then undo that. Target time: 20 s
Solution
k create priorityclass critical --value=1000000 --description="wichtige Apps"
k label node worker1 zone=a
k cordon worker2
k uncordon worker2Expected (excerpt):
priorityclass.scheduling.k8s.io/critical created
node/worker1 labeled
node/worker2 cordoned
node/worker2 uncordoned
cordon sets the taint
node.kubernetes.io/unschedulable:NoSchedule – the same as a
custom taint, just with built-in tools.
Save your progress? Sign up for free
Checkpoint
k get pods -l app=spread -o custom-columns=NODE:.spec.nodeName --no-headers \
| sort | uniq -cExpected:
2 worker1
2 worker2
k describe node worker1 worker2 | grep TaintsExpected: Taints: <none>
twice.
k get pod manual -o jsonpath='{.spec.nodeName}{"\n"}'Expected: worker2
Challenge
Start a Pod prio with the PriorityClass
critical that may only run on nodes with the label
zone=a.
Hint: priorityClassName goes directly
under spec.
Solution
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
name: prio
spec:
priorityClassName: critical
nodeSelector:
zone: a
containers:
- name: prio
image: nginx:1.29
EOF
k get pod prio -o custom-columns=NODE:.spec.nodeName,PRIO:.spec.priorityExpected:
NODE PRIO
worker1 1000000
Clean up:
k config set-context --current --namespace=default
k delete ns tag05
k delete priorityclass critical
k label node worker1 zone-
k label node worker2 disk-Save your progress? Sign up for free
Exam Tip
Pending? Always start withk describe podand read the events at the bottom. The message gives the reason for each node.- Remove a taint: the same line with
-at the end. Remove a label:key-. - Static Pods: work on the right node (
ssh worker1), read the path from/var/lib/kubelet/config.yaml– it may differ in the exam.
Enterprise Note
- Requests without limits for CPU, but always limits for memory – otherwise the neighbors get OOM kills.
- Quotas and LimitRanges per team namespace are part of multi-tenancy.
- Topology spread across
topology.kubernetes.io/zonedistributes Pods across data centers. - For maintenance, use
k draininstead of manual taints (Day 13).