Day 6: Resources and Namespaces
Goal: Every container states what it needs and how much memory it takes at most, and the namespace
noteshas default values and a budget. · Duration: approx. 75 min · Prerequisite: Day 5
Briefing
- Request: What the container is guaranteed to get.
The scheduler only plans with requests – a Pod fits on a node if the sum
of the requests there fits into
allocatable. - Limit: The upper bound. Above the memory limit, the
kernel kills the process (
OOMKilled); above the CPU limit, it is throttled. - Requests and limits determine the QoS class:
Guaranteed,Burstable,BestEffort. It decides who has to go first when memory runs short. - A LimitRange sets default values per container, a ResourceQuota limits the sum in the namespace.
Drill 1: What the Cluster Has
Start of the day:
source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag06 && cd ~/k8s-lab/tag06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag05/notes/." .kubectl get pods -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'
c='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory'
kubectl get nodes -o custom-columns="$c,PODS:.status.allocatable.pods"
kubectl describe node lab-worker | sed -n '/Allocated resources/,/ephemeral/p'Expected (abridged, names and numbers will differ on your machine):
NAME QOS
api-64c6bb58d6-2srkw BestEffort
…
worker-6fc996c95b-d6f58 BestEffort
NAME CPU MEMORY PODS
lab-control-plane 4 8131784Ki 110
lab-worker 4 8131784Ki 110
lab-worker2 4 8131784Ki 110
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (2%) 0 (0%)
memory 50Mi (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
All notes Pods are BestEffort: no requests, no limits.
The scheduler counts them as zero; only kindnet from
kube-system has registered anything. And every kind node
reports the whole Colima VM as its capacity – in
reality, three nodes share the same 4 CPUs. In the lab that doesn’t
matter; in production every node has its own hardware.
100m means 100 millicores, one tenth of a CPU. Memory is
counted in binary prefixes: 64Mi is 64 × 2^20 bytes,
64M would be 64 × 10^6.
Save your progress? Sign up for free
Drill 2: LimitRange and ResourceQuota
The namespace gets default values and a budget. The file
namespace.yaml grows:
cat > k8s/namespace.yaml <<'EOF'
# k8s/namespace.yaml – state at end of Day 6
apiVersion: v1
kind: Namespace
metadata:
name: notes
---
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
namespace: notes
spec:
limits:
- type: Container
defaultRequest:
cpu: 50m
memory: 64Mi
default:
memory: 128Mi
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: notes
namespace: notes
spec:
hard:
pods: "20"
requests.cpu: "2"
requests.memory: 2Gi
limits.memory: 4Gi
EOF
kubectl apply --dry-run=server -f k8s/namespace.yaml
kubectl apply -f k8s/namespace.yaml
kubectl describe limitrange defaults
kubectl describe quota notesExpected:
namespace/notes unchanged (server dry run)
limitrange/defaults created (server dry run)
resourcequota/notes created (server dry run)
namespace/notes unchanged
limitrange/defaults created
resourcequota/notes created
Name: defaults
Namespace: notes
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container memory - - 64Mi 128Mi -
Container cpu - - 50m - -
Name: notes
Namespace: notes
Resource Used Hard
-------- ---- ----
limits.memory 0 4Gi
pods 6 20
requests.cpu 0 2
requests.memory 0 2Gi
default is the default limit,
defaultRequest the default request. The
LimitRange deliberately sets no CPU limit (Drill 7). The running Pods
are already counted (pods 6), but their requests as zero:
LimitRange and quota only check when a Pod is created; existing ones
stay untouched.
Save your progress? Sign up for free
Drill 3: Defaults Take Effect on Creation
worker has no resources: yet. A new Pod is enough:
kubectl rollout restart deploy/worker >/dev/null
kubectl rollout status deploy/worker >/dev/null
sleep 5 # until the old Pod is gone
w=$(kubectl get pod -l app.kubernetes.io/name=worker -o name)
kubectl get "$w" -o jsonpath='{.spec.containers[0].resources}'; echo
kubectl get "$w" -o jsonpath='{.metadata.annotations.kubernetes\.io/limit-ranger}'
echo; kubectl get "$w" -o jsonpath='{.status.qosClass}'; echoExpected:
{"limits":{"memory":"128Mi"},"requests":{"cpu":"50m","memory":"64Mi"}}
LimitRanger plugin set: cpu, memory request for container worker; memory limit for
container worker
Burstable
The LimitRange entered the default values into the Pod on creation
and noted that in an annotation. Without a LimitRange, a quota on
requests.cpu would reject every Pod
without requests – the two belong together.
Save your progress? Sign up for free
Drill 4: Requests and Limits for the App
Now the three Deployments get their own values. api excerpt (web and worker likewise, complete files in the companion code):
# k8s/api.yaml (excerpt) – state at end of Day 6
resources:
requests:
cpu: 50m
memory: 32Mi
limits:
memory: 64Mi| Service | requests.cpu |
requests.memory |
limits.memory |
|---|---|---|---|
| api | 50m | 32Mi | 64Mi |
| web | 50m | 96Mi | 192Mi |
| worker | 20m | 64Mi | 128Mi |
cp "$LAB/code/tag06/notes/k8s/"{api,web,worker}.yaml k8s/
kubectl apply --dry-run=server -f k8s/ | grep configured
kubectl apply -f k8s/ | grep configured
for d in api web worker; do kubectl rollout status deploy/$d >/dev/null; done; sleep 8
kubectl get pods -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'
kubectl get quota notes
kubectl describe node lab-worker | sed -n '/Allocated resources/,/memory/p'Expected (abridged):
deployment.apps/api configured (server dry run)
deployment.apps/web configured (server dry run)
deployment.apps/worker configured (server dry run)
deployment.apps/api configured
deployment.apps/web configured
deployment.apps/worker configured
NAME QOS
api-847756b757-cnws2 Burstable
…
worker-5d4f7f4dbd-jx6hz Burstable
NAME REQUEST LIMIT
notes pods: 6/20, requests.cpu: 270m/2, requests.memory: 352Mi/2Gi limits.memory: …
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 250m (6%) 0 (0%)
memory 210Mi (2%) 320Mi (4%)
requests.cpu 270m = 3 × 50m (api) + 2 × 50m (web) + 20m
(worker). During a rollout, the old Pods are counted until they are
really gone – then it briefly shows pods: 7/20. That is why
a quota needs headroom for maxSurge.
| QoS | Condition | under memory pressure on the node |
|---|---|---|
Guaranteed |
every container: requests = limits for CPU and memory | last |
Burstable |
at least one request or limit, but not Guaranteed | in between |
BestEffort |
no requests, no limits | first |
Save your progress? Sign up for free
Drill 5: OOMKilled Live
A test Pod that eats memory: tail /dev/zero searches for
a line ending that never comes, buffering everything along the way.
cat > oom.yaml <<'EOF'
# oom.yaml – test Pod: eats memory up to the limit
apiVersion: v1
kind: Pod
metadata:
name: oom
spec:
restartPolicy: Never
containers:
- name: oom
image: busybox:1.37
command: [sh, -c, "echo start; tail /dev/zero"]
resources:
requests:
memory: 16Mi
limits:
memory: 32Mi
EOF
kubectl apply -f oom.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Failed pod/oom --timeout 120s
kubectl get pod oom
kubectl describe pod oom | grep -A2 -E '^\s+State:'
kubectl get pod oom -o jsonpath='{.status.qosClass} {.spec.containers[0].resources}'
echo; kubectl delete pod oomExpected (abridged):
pod/oom created
pod/oom condition met
NAME READY STATUS RESTARTS AGE
oom 0/1 OOMKilled 0 8s
State: Terminated
Reason: OOMKilled
Exit Code: 137
Burstable {"limits":{"memory":"32Mi"},"requests":{"cpu":"50m","memory":"16Mi"}}
pod "oom" deleted from notes namespace
Exit 137 = 128 + 9: SIGKILL from the kernel, no chance to clean up.
The LimitRange added the missing CPU request (50m). In a
Deployment (restartPolicy: Always) the kubelet would
restart it – and after a few rounds you would see
CrashLoopBackOff, with Last State: OOMKilled
in kubectl describe.
Save your progress? Sign up for free
Drill 6: Too Big – Quota and
Pending
A Pod that requests 5 CPUs:
cat > big.yaml <<'EOF'
# big.yaml – test Pod with requests that are too big
apiVersion: v1
kind: Pod
metadata:
name: big
spec:
containers:
- name: big
image: busybox:1.37
command: [sleep, "3600"]
resources:
requests:
cpu: "5"
memory: 64Mi
limits:
memory: 64Mi
EOF
kubectl apply -f big.yamlExpected:
Error from server (Forbidden): error when creating "big.yaml": pods "big" is forbidden:
exceeded quota: notes, requested: requests.cpu=5, used: requests.cpu=270m, limited:
requests.cpu=2
The quota already rejects it at the API server; the Pod isn’t even
created. In the namespace default there is no quota – there
the Pod gets through and meets the scheduler:
kubectl apply -n default -f big.yaml
sleep 5
kubectl get pod big -n default
kubectl describe pod big -n default | sed -n '/^Events/,$p'
kubectl delete pod big -n defaultExpected (abridged):
pod/big created
NAME READY STATUS RESTARTS AGE
big 0/1 Pending 0 6s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 6s default-scheduler 0/3 nodes are available: 1 node(s)
had untolerated taint(s), 2 Insufficient cpu. preemption: 0/3 nodes are available: 3
Preemption is not helpful for scheduling.
pod "big" deleted from default namespace
No node has 5 CPUs free, and the control plane node is closed to
normal Pods by a taint. The Pod stays Pending until room
becomes available – forever, if need be. Insufficient cpu
or Insufficient memory almost always means: requests too
big.
Save your progress? Sign up for free
Drill 7: Guaranteed and Why No CPU Limit
A Pod with requests = limits for CPU and memory that wants to max out one CPU:
cat > cpu.yaml <<'EOF'
# cpu.yaml – test Pod: endless loop with a CPU limit
apiVersion: v1
kind: Pod
metadata:
name: cpu
spec:
containers:
- name: cpu
image: busybox:1.37
command: [sh, -c, "while :; do :; done"]
resources:
requests:
cpu: 100m
memory: 16Mi
limits:
cpu: 100m
memory: 16Mi
EOF
kubectl apply -f cpu.yaml && kubectl wait --for=condition=Ready pod/cpu
kubectl get pod cpu -o jsonpath='{.status.qosClass}'; echo
kubectl exec cpu -- cat /sys/fs/cgroup/cpu.max
sleep 10
kubectl exec cpu -- grep -E 'usage_usec|nr_periods|nr_throttled|throttled_usec' \
/sys/fs/cgroup/cpu.stat
kubectl delete pod cpuExpected (abridged, numbers differ):
pod/cpu created
pod/cpu condition met
Guaranteed
10000 100000
usage_usec 1530449
nr_periods 149
nr_throttled 149
throttled_usec 5468460
pod "cpu" deleted from notes namespace
cpu.max is how the limit is implemented in the cgroup:
10 ms of CPU time per 100 ms. The Pod ran for about 15 s and got 1.5 s
of CPU – throttled in every period, even though the
nodes had almost nothing to do.
That is why the notes app only has a memory limit. A CPU request is enough: it secures the share under contention, and the container may use unused CPU too. A CPU limit, by contrast, slows it down even when the node is idle – visible as long response times, never as an error message. Memory is different: without a limit, a leak can hit the whole node.
Save your progress? Sign up for free
Checkpoint
kubectl get limitrangeExpected: the LimitRange defaults.
NAME CREATED AT
defaults 2026-…
kubectl describe quota notesExpected: a quota with 20 Pods; 6 Pods and 270m of CPU
requests are in use.
Name: notes
Namespace: notes
Resource Used Hard
-------- ---- ----
limits.memory 704Mi 4Gi
pods 6 20
requests.cpu 270m 2
requests.memory 352Mi 2Gi
kubectl get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClassExpected: all Pods Burstable.
NAME QOS
api-9cfb78c75-4xsz9 Burstable
api-9cfb78c75-9fj7l Burstable
api-9cfb78c75-m9xxr Burstable
web-857bcf49dc-bs5wm Burstable
web-857bcf49dc-wd9gt Burstable
worker-5c46fbcb8-5t879 Burstable
kubectl describe deploy api | grep -A2 LimitsExpected: only a memory limit, no CPU limit.
Limits:
memory: 64Mi
Requests:
kubectl get pod oom cpu bigExpected: the test Pods from the drills have been cleaned up.
Error from server (NotFound): pods "oom" not found
Error from server (NotFound): pods "cpu" not found
Error from server (NotFound): pods "big" not found
Challenge
Scale api to 30 replicas. How many are running in the end, and where do you find the reason for the rest? Then scale back to 3.
Hint: It isn’t the Deployment that creates Pods but its ReplicaSet – and the quota counts more than just CPU.
Solution
kubectl scale deploy api --replicas 30
sleep 30
kubectl get deploy api
kubectl get quota notes
kubectl get deploy api \
-o jsonpath='{.status.conditions[?(@.type=="ReplicaFailure")].message}'; echo
kubectl scale deploy api --replicas 3Expected (abridged):
deployment.apps/api scaled
NAME READY UP-TO-DATE AVAILABLE AGE
api 17/30 17 17 23m
NAME REQUEST LIMIT
notes pods: 20/20, requests.cpu: 970m/2, requests.memory: 800Mi/2Gi limits.memory: …
pods "api-847756b757-q6dj4" is forbidden: exceeded quota: notes, requested: pods=1,
used: pods=20, limited: pods=20
deployment.apps/api scaled
At 20 Pods it stops: 17 api, 2 web, 1 worker. The ReplicaSet keeps
trying and reports FailedCreate in its events; the
Deployment sums up the reason in the condition
ReplicaFailure. By the way, a
kubectl apply -f k8s/ would also have reset it –
replicas: 3 is in the file.
Save your progress? Sign up for free
Enterprise Note
- Set requests based on measurements, not gut feeling: metrics-server
plus
kubectl top, Prometheus or the Vertical Pod Autoscaler in recommendation mode. - Quotas per team namespace are the usual way to separate tenants;
also limit
ephemeral-storage, otherwise a log fills the node disk. - The kubelet reserves room for the system with
kube-reservedandsystem-reserved; without this reserve, a Pod memory leak takes the node down with it. - PriorityClasses decide who gets evicted when room runs short – important for monitoring and ingress Pods.
- CPU limits are mandatory in some environments (billing, tenants).
Then set them generously and watch the throttling
(
container_cpu_cfs_throttled_periods_total).