Day 6: Autoscaling and Monitoring
Goal: You see what nodes and Pods consume and let Kubernetes scale Pods by load – horizontally and in size. · Duration: approx. 45 min · Prerequisite: Day 5
Briefing
CKA: “Configure workload autoscaling” (Workloads) and “Monitor cluster and application resource usage” (Troubleshooting).
kubelet (cAdvisor) --> metrics-server --> Metrics API --> kubectl top
\--> HPA --> Deployment.replicas
- Without metrics-server, no
kubectl topand no HPA. - The HPA calculates in percent of the requests – no CPU request, no scaling.
- In-place resize changes the requests and limits of a running Pod without a restart.
Drill 1: Installing metrics-server
M=https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.1
k apply -f $M/components.yamlExpected (last lines):
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
kubeadm issues self-signed serving certificates to the kubelets, which metrics-server does not trust. In the lab, we turn off the verification:
k patch deployment metrics-server -n kube-system --type=json \
-p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
"value":"--kubelet-insecure-tls"}]'
k rollout status deployment metrics-server -n kube-system
k wait --for=condition=Available apiservice v1beta1.metrics.k8s.ioExpected:
deployment.apps/metrics-server patched
deployment "metrics-server" successfully rolled out
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io condition met
Warning:
--kubelet-insecure-tlsonly in the lab. In production, use signed kubelet certificates (Enterprise Note).
Save your progress? Sign up for free
Drill 2: Who Consumes What?
After about 30 seconds, the first values are available:
k top nodes
k top pods -n kube-system --sort-by=memory | head -4Expected:
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
cp1 615m 15% 1235Mi 10%
worker1 410m 20% 1072Mi 28%
worker2 280m 14% 709Mi 18%
NAME CPU(cores) MEMORY(bytes)
kube-apiserver-cp1 240m 492Mi
kube-controller-manager-cp1 44m 70Mi
etcd-cp1 93m 58Mi
Per container instead of per Pod:
k top pods -n kube-system --containers | head -3Expected:
POD NAME CPU(cores) MEMORY(bytes)
coredns-589f44dc88-c9tfc coredns 8m 12Mi
coredns-589f44dc88-tdzsw coredns 7m 12Mi
Save your progress? Sign up for free
Drill 3: An App with CPU Requests
The example image from the Kubernetes docs performs a computation on every request:
k create ns tag06
k config set-context --current --namespace=tag06
k create deployment php --image=registry.k8s.io/hpa-example --port=80
k set resources deployment php --requests=cpu=200m --limits=cpu=500m
k expose deployment php --port=80Expected:
deployment.apps/php created
deployment.apps/php resource requirements updated
service/php exposed
Save your progress? Sign up for free
Drill 4: Horizontal Pod Autoscaler
k autoscale deployment php --cpu=50% --min=1 --max=5
k get hpa phpExpected:
horizontalpodautoscaler.autoscaling/php autoscaled
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php Deployment/php cpu: <unknown>/50% 1 5 1 68s
<unknown> disappears after one to two minutes, as
soon as metrics are available. Until then,
k describe hpa php reports events such as
FailedGetResourceMetric – that is normal.
The generated object is an autoscaling/v2 HPA:
k autoscale deployment php --cpu=50% --min=1 --max=5 $do | head -14Expected:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: php
spec:
maxReplicas: 5
metrics:
- resource:
name: cpu
target:
averageUtilization: 50
type: Utilization
type: Resource
minReplicas: 1
Save your progress? Sign up for free
Drill 5: Generating Load
k run load --image=busybox:1.37 --restart=Never -- \
sh -c 'while true; do wget -q -O- http://php >/dev/null; done'After about 90 seconds:
k get hpa php
k top podsExpected (values fluctuate):
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php Deployment/php cpu: 65%/50% 1 5 5 2m45s
NAME CPU(cores) MEMORY(bytes)
load 11m 0Mi
php-6b569995f-5kmct 219m 11Mi
php-6b569995f-6gwhf 20m 11Mi
php-6b569995f-bwtjd 129m 10Mi
php-6b569995f-cfqrh 149m 12Mi
php-6b569995f-dtdsk 58m 11Mi
Five replicas – the maximum. Stop the load:
k delete pod load --nowSave your progress? Sign up for free
Drill 6: Scaling Down Faster
By default, the HPA waits five minutes before removing Pods – protection against flapping. For the lab, we shorten the window:
k patch hpa php \
-p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":30}}}}'
k get hpa php -o jsonpath='{.spec.behavior.scaleDown}{"\n"}'Expected:
horizontalpodautoscaler.autoscaling/php patched
{"policies":[{"periodSeconds":15,"type":"Percent","value":100}],"selectPolicy":"Max",
"stabilizationWindowSeconds":30}
After a little over a minute:
k get hpa phpExpected: REPLICAS is back at
1.
Save your progress? Sign up for free
Drill 7: In-Place Resize
Since Kubernetes 1.35, the resize subresource changes
the CPU and memory of a running Pod:
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
name: resize
spec:
containers:
- name: app
image: nginx:1.29
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
EOF
k wait --for=condition=Ready pod/resize
k patch pod resize --subresource resize -p '{"spec":{"containers":[{"name":"app",
"resources":{"requests":{"cpu":"300m"},"limits":{"cpu":"500m"}}}]}}'
k get pod resize -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}'
k get pod resize -o jsonpath='{.status.containerStatuses[0].restartCount}{"\n"}'Expected:
pod/resize patched
{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"300m","memory":"64Mi"}}
0
The new values apply, and the container was not restarted. A normal
k apply with changed resources, on the other hand, would be
rejected (Day 2).
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the node with the highest CPU load first. Target time: 15 s
- Find the Pod in the whole cluster that uses the most memory and
write its name to
/tmp/top-pod.txt. Target time: 60 s - Create an HPA for
php: 2 to 8 replicas, target 70 % CPU. Target time: 20 s
Solution
k top node --sort-by=cpu
k top pods -A --sort-by=memory --no-headers | head -1 \
| awk '{print $2}' > /tmp/top-pod.txt
cat /tmp/top-pod.txt
k delete hpa php
k autoscale deployment php --cpu=70% --min=2 --max=8Expected (excerpt):
kube-apiserver-cp1
horizontalpodautoscaler.autoscaling/php autoscaled
Save your progress? Sign up for free
Checkpoint
k get apiservice v1beta1.metrics.k8s.ioExpected: AVAILABLE is
True.
k get hpa php -o jsonpath='{.spec.minReplicas} {.spec.maxReplicas}{"\n"}'Expected: 2 8
k get pod resize -o jsonpath='{.spec.containers[0].resources.requests.cpu}{"\n"}'Expected: 300m
Challenge
The memory limit of the Pod resize should go up to 256Mi
– without a restart.
Hint: Memory changes go through the same subresource as CPU.
Solution
k patch pod resize --subresource resize \
-p '{"spec":{"containers":[{"name":"app",
"resources":{"limits":{"memory":"256Mi"}}}]}}'
k get pod resize -o jsonpath='{.status.containerStatuses[0].resources.limits}{"\n"}'
k get pod resize -o jsonpath='{.status.containerStatuses[0].restartCount}{"\n"}'Expected:
pod/resize patched
{"cpu":"500m","memory":"256Mi"}
0
Clean up (metrics-server stays for the following days):
k config set-context --current --namespace=default
k delete ns tag06Save your progress? Sign up for free
Exam Tip
k topis the fastest answer to “which Pod uses the most” – with--sort-byand-A.- HPA stuck at
<unknown>? Either the CPU request is missing or metrics-server is not running. - Remember this docs page: “HorizontalPodAutoscaler Walkthrough”.
Enterprise Note
- Instead of
--kubelet-insecure-tls: setserverTLSBootstrap: truein the kubelet and approve the serving CSRs (or have them approved) – then metrics-server verifies real certificates. kubectl toponly shows the current moment. For history and alerts, use Prometheus (book “Observability hands-on”).- For event-driven scaling (queues, Kafka), use KEDA; for nodes, Cluster Autoscaler or Karpenter.