Skip to content
Hands-on IT IT Drills

Day 5: Scheduling

Goal: You control where Pods land and how much they are allowed to consume – and you see at once why a Pod stays Pending. · Duration: approx. 75 min · Prerequisite: Day 4

Briefing

CKA: “Configure Pod admission and scheduling (limits, node affinity, etc.)” (part of the 15 %).

  • Requests decide placement, limits set the upper bound at runtime.
  • LimitRange sets default values, ResourceQuota caps a namespace.
  • nodeSelector / affinity pull Pods toward nodes, taints repel them, tolerations allow the exception.
  • If a Pod stays Pending, the reason is in the events (FailedScheduling).
mkdir -p ~/cka/tag05 && cd ~/cka/tag05
k create ns tag05
k config set-context --current --namespace=tag05

Drill 1: A Pod That Fits Nowhere

How much of worker1 is already allocated?

k describe node worker1 | grep -A6 'Allocated resources'

Expected (abridged):

Allocated resources:
  Resource           Requests    Limits
  --------           --------    ------
  cpu                200m (10%)  0 (0%)
  memory             140Mi (3%)  340Mi (8%)

Now a Pod that requests 8 CPUs – generated and given resources in a single pipe:

k run big --image=nginx:1.29 $do \
  | k set resources --local -f - --requests=cpu=8 -o yaml | k apply -f -
k get pod big
k get events --field-selector involvedObject.name=big | tail -1

Expected (abridged):

NAME   READY   STATUS    RESTARTS   AGE
big    0/1     Pending   0          6s
Warning   FailedScheduling   pod/big   0/3 nodes are available: 1 node(s) had
untolerated taint(s), 2 Insufficient cpu. ...

Read the message carefully: cp1 is ruled out by the taint, the workers by the CPU.

k delete pod big --now

Save your progress? Sign up for free

Drill 2: LimitRange and ResourceQuota

k create quota tag05-quota \
  --hard=pods=4,requests.cpu=1,requests.memory=1Gi,limits.memory=2Gi
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
spec:
  limits:
  - type: Container
    default:
      memory: 256Mi
    defaultRequest:
      cpu: 100m
      memory: 128Mi
EOF
k run small --image=nginx:1.29
k get pod small -o jsonpath='{.spec.containers[0].resources}{"\n"}'

Expected:

{"limits":{"memory":"256Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}

The Pod had no resources – the LimitRange filled them in. Without it, the quota would reject the Pod, because with requests.cpu in the quota, every Pod must have requests.

k describe quota tag05-quota | tail -6
k create deployment many --image=nginx:1.29 --replicas=6
k get deployment many

Expected:

Resource         Used   Hard
--------         ----   ----
limits.memory    256Mi  2Gi
pods             1      4
requests.cpu     100m   1
requests.memory  128Mi  1Gi
deployment.apps/many created
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
many   3/6     3            3           8s

Only three new Pods – together with small, the quota of 4 is full. You find the reason on the ReplicaSet, not on the Deployment:

k describe rs -l app=many | grep -m1 'exceeded quota'

Expected (abridged):

  Warning  FailedCreate  8s  replicaset-controller  Error creating:
pods "many-6df65cc865-lgb8s" is forbidden: exceeded quota: tag05-quota,
requested: pods=1, used: pods=4, limited: pods=4

Clean up:

k delete deployment many
k delete pod small
k delete quota tag05-quota
k delete limitrange defaults

Save your progress? Sign up for free

Drill 3: nodeSelector and Node Affinity

k label node worker2 disk=ssd
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: fast
spec:
  nodeSelector:
    disk: ssd
  containers:
  - name: fast
    image: nginx:1.29
---
apiVersion: v1
kind: Pod
metadata:
  name: affine
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disk
            operator: NotIn
            values: ["ssd"]
          - key: node-role.kubernetes.io/control-plane
            operator: DoesNotExist
  containers:
  - name: affine
    image: nginx:1.29
EOF
k get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Expected:

NAME     NODE
affine   worker1
fast     worker2

Expressions within one matchExpressions are combined with AND, multiple nodeSelectorTerms with OR. preferredDuringScheduling… would only be a preference, not a requirement.

k delete pod fast affine --now

Save your progress? Sign up for free

Drill 4: Taints and Tolerations

worker1 goes into maintenance – new Pods should avoid it:

k taint node worker1 maintenance=true:NoSchedule
k create deployment spread --image=nginx:1.29 --replicas=3
k rollout status deployment spread
k get pods -l app=spread -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Expected:

NAME                      NODE
spread-85cb54f4b9-fr7nl   worker2
spread-85cb54f4b9-lkjq6   worker2
spread-85cb54f4b9-w8wgl   worker2

A Pod with a matching toleration may still go there:

cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: tolerant
spec:
  tolerations:
  - key: maintenance
    operator: Equal
    value: "true"
    effect: NoSchedule
  nodeSelector:
    kubernetes.io/hostname: worker1
  containers:
  - name: tolerant
    image: nginx:1.29
EOF
k get pod tolerant -o wide

Expected: tolerant runs on worker1. The toleration allows the node but does not force it – here the nodeSelector takes care of that.

Remove the taint again (the - at the end):

k taint node worker1 maintenance=true:NoSchedule-
k describe node worker1 | grep Taints

Expected:

node/worker1 untainted
Taints:             <none>

Save your progress? Sign up for free

Drill 5: NoExecute Clears the Node

NoSchedule leaves running Pods alone. NoExecute evicts them:

k taint node worker2 evict=now:NoExecute
k get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Expected (after about 10 s):

NAME                      NODE
spread-85cb54f4b9-4vczg   worker1
spread-85cb54f4b9-6fsfr   worker1
spread-85cb54f4b9-nn2dh   worker1
tolerant                  worker1

All three spread Pods have moved to worker1.

k taint node worker2 evict=now:NoExecute-
k delete deployment spread
k delete pod tolerant --now

Save your progress? Sign up for free

Drill 6: Spreading Pods – and a Trap

cat > spread.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spread
spec:
  replicas: 4
  selector:
    matchLabels:
      app: spread
  template:
    metadata:
      labels:
        app: spread
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: spread
      containers:
      - name: nginx
        image: nginx:1.29
EOF
k apply -f spread.yaml
k get pods -l app=spread -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Expected:

NAME                      NODE
spread-867df78bcb-ssvvs   <none>
spread-867df78bcb-b4fl7   <none>
spread-867df78bcb-mhqbk   worker1
spread-867df78bcb-vs768   worker2

Two Pods stay Pending. Why?

k get events --field-selector reason=FailedScheduling | tail -1

Expected (abridged):

Warning   FailedScheduling   pod/spread-867df78bcb-ssvvs   0/3 nodes are available:
1 node(s) had untolerated taint(s), 2 node(s) didn't match pod topology spread
constraints. ...

The scheduler counts cp1 as a topology domain with 0 Pods, even though no Pod can ever land there because of the taint. With 0 on cp1, maxSkew: 1 allows only one Pod per worker. The fix: take taints into account when spreading.

sed -i \
  's/^        whenUnsatisfiable: DoNotSchedule/&\n        nodeTaintsPolicy: Honor/' \
  spread.yaml
grep -A1 whenUnsatisfiable spread.yaml
k delete deployment spread
k apply -f spread.yaml
k rollout status deployment spread
k get pods -l app=spread -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

Expected:

        whenUnsatisfiable: DoNotSchedule
        nodeTaintsPolicy: Honor
deployment.apps "spread" deleted from tag05 namespace
deployment.apps/spread created
deployment "spread" successfully rolled out
      2 worker1
      2 worker2

Why delete instead of running apply on the running Deployment? Topology spread only takes effect at scheduling time. During a rolling update, the old Pods are still counted, and the distribution stays lopsided.

Save your progress? Sign up for free

Drill 7: Static Pod and Manual Scheduling

A static Pod runs without the scheduler – the kubelet starts it from a file. On worker1:

cat <<'EOF' | sudo tee /etc/kubernetes/manifests/static-web.yaml
apiVersion: v1
kind: Pod
metadata:
  name: static-web
  namespace: tag05
spec:
  containers:
  - name: nginx
    image: nginx:1.29
EOF

Back on cp1:

k get pods | grep static
k delete pod static-web-worker1
k get pods | grep static

Expected:

static-web-worker1        1/1     Running   0          16s
pod "static-web-worker1" deleted from tag05 namespace
static-web-worker1        0/1     Pending   0          6s

The Pod is back immediately – its name gets the node as a suffix, and it can only be deleted through the file. On worker1:

sudo rm /etc/kubernetes/manifests/static-web.yaml

You can also bypass the scheduler with nodeName:

cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: manual
spec:
  nodeName: worker2
  containers:
  - name: manual
    image: nginx:1.29
EOF
k get pod manual -o wide

Expected: manual runs on worker2. nodeName ignores affinity and even NoSchedule taints – useful when the scheduler itself is broken (Day 14).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Create a PriorityClass critical with the value 1000000. Target time: 20 s
  2. Label worker1 with zone=a. Target time: 10 s
  3. Make worker2 unusable for new Pods without disturbing running Pods, then undo that. Target time: 20 s

Solution

k create priorityclass critical --value=1000000 --description="wichtige Apps"
k label node worker1 zone=a
k cordon worker2
k uncordon worker2

Expected (excerpt):

priorityclass.scheduling.k8s.io/critical created
node/worker1 labeled
node/worker2 cordoned
node/worker2 uncordoned

cordon sets the taint node.kubernetes.io/unschedulable:NoSchedule – the same as a custom taint, just with built-in tools.

Save your progress? Sign up for free

Checkpoint

k get pods -l app=spread -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

Expected:

      2 worker1
      2 worker2
k describe node worker1 worker2 | grep Taints

Expected: Taints: <none> twice.

k get pod manual -o jsonpath='{.spec.nodeName}{"\n"}'

Expected: worker2

Challenge

Start a Pod prio with the PriorityClass critical that may only run on nodes with the label zone=a.

Hint: priorityClassName goes directly under spec.

Solution

cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: prio
spec:
  priorityClassName: critical
  nodeSelector:
    zone: a
  containers:
  - name: prio
    image: nginx:1.29
EOF
k get pod prio -o custom-columns=NODE:.spec.nodeName,PRIO:.spec.priority

Expected:

NODE      PRIO
worker1   1000000

Clean up:

k config set-context --current --namespace=default
k delete ns tag05
k delete priorityclass critical
k label node worker1 zone-
k label node worker2 disk-

Save your progress? Sign up for free

Exam Tip

  • Pending? Always start with k describe pod and read the events at the bottom. The message gives the reason for each node.
  • Remove a taint: the same line with - at the end. Remove a label: key-.
  • Static Pods: work on the right node (ssh worker1), read the path from /var/lib/kubelet/config.yaml – it may differ in the exam.

Enterprise Note

  • Requests without limits for CPU, but always limits for memory – otherwise the neighbors get OOM kills.
  • Quotas and LimitRanges per team namespace are part of multi-tenancy.
  • Topology spread across topology.kubernetes.io/zone distributes Pods across data centers.
  • For maintenance, use k drain instead of manual taints (Day 13).