Skip to content
Hands-on IT IT Drills

Day 6: Resources and Namespaces

Goal: Every container states what it needs and how much memory it takes at most, and the namespace notes has default values and a budget. · Duration: approx. 75 min · Prerequisite: Day 5

Briefing

  • Request: What the container is guaranteed to get. The scheduler only plans with requests – a Pod fits on a node if the sum of the requests there fits into allocatable.
  • Limit: The upper bound. Above the memory limit, the kernel kills the process (OOMKilled); above the CPU limit, it is throttled.
  • Requests and limits determine the QoS class: Guaranteed, Burstable, BestEffort. It decides who has to go first when memory runs short.
  • A LimitRange sets default values per container, a ResourceQuota limits the sum in the namespace.

Drill 1: What the Cluster Has

Start of the day:

source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag06 && cd ~/k8s-lab/tag06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag05/notes/." .
kubectl get pods -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'
c='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory'
kubectl get nodes -o custom-columns="$c,PODS:.status.allocatable.pods"
kubectl describe node lab-worker | sed -n '/Allocated resources/,/ephemeral/p'

Expected (abridged, names and numbers will differ on your machine):

NAME                      QOS
api-64c6bb58d6-2srkw      BestEffort
…
worker-6fc996c95b-d6f58   BestEffort
NAME                 CPU   MEMORY      PODS
lab-control-plane    4     8131784Ki   110
lab-worker           4     8131784Ki   110
lab-worker2          4     8131784Ki   110
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (2%)  0 (0%)
  memory             50Mi (0%)  0 (0%)
  ephemeral-storage  0 (0%)     0 (0%)

All notes Pods are BestEffort: no requests, no limits. The scheduler counts them as zero; only kindnet from kube-system has registered anything. And every kind node reports the whole Colima VM as its capacity – in reality, three nodes share the same 4 CPUs. In the lab that doesn’t matter; in production every node has its own hardware.

100m means 100 millicores, one tenth of a CPU. Memory is counted in binary prefixes: 64Mi is 64 × 2^20 bytes, 64M would be 64 × 10^6.

Save your progress? Sign up for free

Drill 2: LimitRange and ResourceQuota

The namespace gets default values and a budget. The file namespace.yaml grows:

cat > k8s/namespace.yaml <<'EOF'
# k8s/namespace.yaml – state at end of Day 6
apiVersion: v1
kind: Namespace
metadata:
  name: notes
---
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: notes
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 50m
        memory: 64Mi
      default:
        memory: 128Mi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: notes
  namespace: notes
spec:
  hard:
    pods: "20"
    requests.cpu: "2"
    requests.memory: 2Gi
    limits.memory: 4Gi
EOF
kubectl apply --dry-run=server -f k8s/namespace.yaml
kubectl apply -f k8s/namespace.yaml
kubectl describe limitrange defaults
kubectl describe quota notes

Expected:

namespace/notes unchanged (server dry run)
limitrange/defaults created (server dry run)
resourcequota/notes created (server dry run)
namespace/notes unchanged
limitrange/defaults created
resourcequota/notes created
Name:       defaults
Namespace:  notes
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   memory    -    -    64Mi             128Mi          -
Container   cpu       -    -    50m              -              -
Name:            notes
Namespace:       notes
Resource         Used  Hard
--------         ----  ----
limits.memory    0     4Gi
pods             6     20
requests.cpu     0     2
requests.memory  0     2Gi

default is the default limit, defaultRequest the default request. The LimitRange deliberately sets no CPU limit (Drill 7). The running Pods are already counted (pods 6), but their requests as zero: LimitRange and quota only check when a Pod is created; existing ones stay untouched.

Save your progress? Sign up for free

Drill 3: Defaults Take Effect on Creation

worker has no resources: yet. A new Pod is enough:

kubectl rollout restart deploy/worker >/dev/null
kubectl rollout status deploy/worker >/dev/null
sleep 5   # until the old Pod is gone
w=$(kubectl get pod -l app.kubernetes.io/name=worker -o name)
kubectl get "$w" -o jsonpath='{.spec.containers[0].resources}'; echo
kubectl get "$w" -o jsonpath='{.metadata.annotations.kubernetes\.io/limit-ranger}'
echo; kubectl get "$w" -o jsonpath='{.status.qosClass}'; echo

Expected:

{"limits":{"memory":"128Mi"},"requests":{"cpu":"50m","memory":"64Mi"}}
LimitRanger plugin set: cpu, memory request for container worker; memory limit for
container worker
Burstable

The LimitRange entered the default values into the Pod on creation and noted that in an annotation. Without a LimitRange, a quota on requests.cpu would reject every Pod without requests – the two belong together.

Save your progress? Sign up for free

Drill 4: Requests and Limits for the App

Now the three Deployments get their own values. api excerpt (web and worker likewise, complete files in the companion code):

# k8s/api.yaml (excerpt) – state at end of Day 6
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 64Mi
Service requests.cpu requests.memory limits.memory
api 50m 32Mi 64Mi
web 50m 96Mi 192Mi
worker 20m 64Mi 128Mi
cp "$LAB/code/tag06/notes/k8s/"{api,web,worker}.yaml k8s/
kubectl apply --dry-run=server -f k8s/ | grep configured
kubectl apply -f k8s/ | grep configured
for d in api web worker; do kubectl rollout status deploy/$d >/dev/null; done; sleep 8
kubectl get pods -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'
kubectl get quota notes
kubectl describe node lab-worker | sed -n '/Allocated resources/,/memory/p'

Expected (abridged):

deployment.apps/api configured (server dry run)
deployment.apps/web configured (server dry run)
deployment.apps/worker configured (server dry run)
deployment.apps/api configured
deployment.apps/web configured
deployment.apps/worker configured
NAME                      QOS
api-847756b757-cnws2      Burstable
…
worker-5d4f7f4dbd-jx6hz   Burstable
NAME    REQUEST                                                        LIMIT
notes   pods: 6/20, requests.cpu: 270m/2, requests.memory: 352Mi/2Gi   limits.memory: …
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                250m (6%)   0 (0%)
  memory             210Mi (2%)  320Mi (4%)

requests.cpu 270m = 3 × 50m (api) + 2 × 50m (web) + 20m (worker). During a rollout, the old Pods are counted until they are really gone – then it briefly shows pods: 7/20. That is why a quota needs headroom for maxSurge.

QoS Condition under memory pressure on the node
Guaranteed every container: requests = limits for CPU and memory last
Burstable at least one request or limit, but not Guaranteed in between
BestEffort no requests, no limits first

Save your progress? Sign up for free

Drill 5: OOMKilled Live

A test Pod that eats memory: tail /dev/zero searches for a line ending that never comes, buffering everything along the way.

cat > oom.yaml <<'EOF'
# oom.yaml – test Pod: eats memory up to the limit
apiVersion: v1
kind: Pod
metadata:
  name: oom
spec:
  restartPolicy: Never
  containers:
    - name: oom
      image: busybox:1.37
      command: [sh, -c, "echo start; tail /dev/zero"]
      resources:
        requests:
          memory: 16Mi
        limits:
          memory: 32Mi
EOF
kubectl apply -f oom.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Failed pod/oom --timeout 120s
kubectl get pod oom
kubectl describe pod oom | grep -A2 -E '^\s+State:'
kubectl get pod oom -o jsonpath='{.status.qosClass} {.spec.containers[0].resources}'
echo; kubectl delete pod oom

Expected (abridged):

pod/oom created
pod/oom condition met
NAME   READY   STATUS      RESTARTS   AGE
oom    0/1     OOMKilled   0          8s
    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
Burstable {"limits":{"memory":"32Mi"},"requests":{"cpu":"50m","memory":"16Mi"}}
pod "oom" deleted from notes namespace

Exit 137 = 128 + 9: SIGKILL from the kernel, no chance to clean up. The LimitRange added the missing CPU request (50m). In a Deployment (restartPolicy: Always) the kubelet would restart it – and after a few rounds you would see CrashLoopBackOff, with Last State: OOMKilled in kubectl describe.

Save your progress? Sign up for free

Drill 6: Too Big – Quota and Pending

A Pod that requests 5 CPUs:

cat > big.yaml <<'EOF'
# big.yaml – test Pod with requests that are too big
apiVersion: v1
kind: Pod
metadata:
  name: big
spec:
  containers:
    - name: big
      image: busybox:1.37
      command: [sleep, "3600"]
      resources:
        requests:
          cpu: "5"
          memory: 64Mi
        limits:
          memory: 64Mi
EOF
kubectl apply -f big.yaml

Expected:

Error from server (Forbidden): error when creating "big.yaml": pods "big" is forbidden:
exceeded quota: notes, requested: requests.cpu=5, used: requests.cpu=270m, limited:
requests.cpu=2

The quota already rejects it at the API server; the Pod isn’t even created. In the namespace default there is no quota – there the Pod gets through and meets the scheduler:

kubectl apply -n default -f big.yaml
sleep 5
kubectl get pod big -n default
kubectl describe pod big -n default | sed -n '/^Events/,$p'
kubectl delete pod big -n default

Expected (abridged):

pod/big created
NAME   READY   STATUS    RESTARTS   AGE
big    0/1     Pending   0          6s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  6s    default-scheduler  0/3 nodes are available: 1 node(s)
had untolerated taint(s), 2 Insufficient cpu. preemption: 0/3 nodes are available: 3
Preemption is not helpful for scheduling.
pod "big" deleted from default namespace

No node has 5 CPUs free, and the control plane node is closed to normal Pods by a taint. The Pod stays Pending until room becomes available – forever, if need be. Insufficient cpu or Insufficient memory almost always means: requests too big.

Save your progress? Sign up for free

Drill 7: Guaranteed and Why No CPU Limit

A Pod with requests = limits for CPU and memory that wants to max out one CPU:

cat > cpu.yaml <<'EOF'
# cpu.yaml – test Pod: endless loop with a CPU limit
apiVersion: v1
kind: Pod
metadata:
  name: cpu
spec:
  containers:
    - name: cpu
      image: busybox:1.37
      command: [sh, -c, "while :; do :; done"]
      resources:
        requests:
          cpu: 100m
          memory: 16Mi
        limits:
          cpu: 100m
          memory: 16Mi
EOF
kubectl apply -f cpu.yaml && kubectl wait --for=condition=Ready pod/cpu
kubectl get pod cpu -o jsonpath='{.status.qosClass}'; echo
kubectl exec cpu -- cat /sys/fs/cgroup/cpu.max
sleep 10
kubectl exec cpu -- grep -E 'usage_usec|nr_periods|nr_throttled|throttled_usec' \
  /sys/fs/cgroup/cpu.stat
kubectl delete pod cpu

Expected (abridged, numbers differ):

pod/cpu created
pod/cpu condition met
Guaranteed
10000 100000
usage_usec 1530449
nr_periods 149
nr_throttled 149
throttled_usec 5468460
pod "cpu" deleted from notes namespace

cpu.max is how the limit is implemented in the cgroup: 10 ms of CPU time per 100 ms. The Pod ran for about 15 s and got 1.5 s of CPU – throttled in every period, even though the nodes had almost nothing to do.

That is why the notes app only has a memory limit. A CPU request is enough: it secures the share under contention, and the container may use unused CPU too. A CPU limit, by contrast, slows it down even when the node is idle – visible as long response times, never as an error message. Memory is different: without a limit, a leak can hit the whole node.

Save your progress? Sign up for free

Checkpoint

kubectl get limitrange

Expected: the LimitRange defaults.

NAME       CREATED AT
defaults   2026-…
kubectl describe quota notes

Expected: a quota with 20 Pods; 6 Pods and 270m of CPU requests are in use.

Name:            notes
Namespace:       notes
Resource         Used   Hard
--------         ----   ----
limits.memory    704Mi  4Gi
pods             6      20
requests.cpu     270m   2
requests.memory  352Mi  2Gi
kubectl get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass

Expected: all Pods Burstable.

NAME                     QOS
api-9cfb78c75-4xsz9      Burstable
api-9cfb78c75-9fj7l      Burstable
api-9cfb78c75-m9xxr      Burstable
web-857bcf49dc-bs5wm     Burstable
web-857bcf49dc-wd9gt     Burstable
worker-5c46fbcb8-5t879   Burstable
kubectl describe deploy api | grep -A2 Limits

Expected: only a memory limit, no CPU limit.

    Limits:
      memory:  64Mi
    Requests:
kubectl get pod oom cpu big

Expected: the test Pods from the drills have been cleaned up.

Error from server (NotFound): pods "oom" not found
Error from server (NotFound): pods "cpu" not found
Error from server (NotFound): pods "big" not found

Challenge

Scale api to 30 replicas. How many are running in the end, and where do you find the reason for the rest? Then scale back to 3.

Hint: It isn’t the Deployment that creates Pods but its ReplicaSet – and the quota counts more than just CPU.

Solution

kubectl scale deploy api --replicas 30
sleep 30
kubectl get deploy api
kubectl get quota notes
kubectl get deploy api \
  -o jsonpath='{.status.conditions[?(@.type=="ReplicaFailure")].message}'; echo
kubectl scale deploy api --replicas 3

Expected (abridged):

deployment.apps/api scaled
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    17/30   17           17          23m
NAME    REQUEST                                                         LIMIT
notes   pods: 20/20, requests.cpu: 970m/2, requests.memory: 800Mi/2Gi   limits.memory: …
pods "api-847756b757-q6dj4" is forbidden: exceeded quota: notes, requested: pods=1,
used: pods=20, limited: pods=20
deployment.apps/api scaled

At 20 Pods it stops: 17 api, 2 web, 1 worker. The ReplicaSet keeps trying and reports FailedCreate in its events; the Deployment sums up the reason in the condition ReplicaFailure. By the way, a kubectl apply -f k8s/ would also have reset it – replicas: 3 is in the file.

Save your progress? Sign up for free

Enterprise Note

  • Set requests based on measurements, not gut feeling: metrics-server plus kubectl top, Prometheus or the Vertical Pod Autoscaler in recommendation mode.
  • Quotas per team namespace are the usual way to separate tenants; also limit ephemeral-storage, otherwise a log fills the node disk.
  • The kubelet reserves room for the system with kube-reserved and system-reserved; without this reserve, a Pod memory leak takes the node down with it.
  • PriorityClasses decide who gets evicted when room runs short – important for monitoring and ingress Pods.
  • CPU limits are mandatory in some environments (billing, tenants). Then set them generously and watch the throttling (container_cpu_cfs_throttled_periods_total).