Skip to content
Hands-on IT IT Drills

Day 6: Autoscaling and Monitoring

Goal: You see what nodes and Pods consume and let Kubernetes scale Pods by load – horizontally and in size. · Duration: approx. 45 min · Prerequisite: Day 5

Briefing

CKA: “Configure workload autoscaling” (Workloads) and “Monitor cluster and application resource usage” (Troubleshooting).

  kubelet (cAdvisor) --> metrics-server --> Metrics API --> kubectl top
                                                      \--> HPA --> Deployment.replicas
  • Without metrics-server, no kubectl top and no HPA.
  • The HPA calculates in percent of the requests – no CPU request, no scaling.
  • In-place resize changes the requests and limits of a running Pod without a restart.

Drill 1: Installing metrics-server

M=https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.1
k apply -f $M/components.yaml

Expected (last lines):

deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

kubeadm issues self-signed serving certificates to the kubelets, which metrics-server does not trust. In the lab, we turn off the verification:

k patch deployment metrics-server -n kube-system --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
  "value":"--kubelet-insecure-tls"}]'
k rollout status deployment metrics-server -n kube-system
k wait --for=condition=Available apiservice v1beta1.metrics.k8s.io

Expected:

deployment.apps/metrics-server patched
deployment "metrics-server" successfully rolled out
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io condition met

Warning: --kubelet-insecure-tls only in the lab. In production, use signed kubelet certificates (Enterprise Note).

Save your progress? Sign up for free

Drill 2: Who Consumes What?

After about 30 seconds, the first values are available:

k top nodes
k top pods -n kube-system --sort-by=memory | head -4

Expected:

NAME      CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
cp1       615m         15%      1235Mi          10%
worker1   410m         20%      1072Mi          28%
worker2   280m         14%      709Mi           18%
NAME                              CPU(cores)   MEMORY(bytes)
kube-apiserver-cp1                240m         492Mi
kube-controller-manager-cp1       44m          70Mi
etcd-cp1                          93m          58Mi

Per container instead of per Pod:

k top pods -n kube-system --containers | head -3

Expected:

POD                               NAME                      CPU(cores)   MEMORY(bytes)
coredns-589f44dc88-c9tfc          coredns                   8m           12Mi
coredns-589f44dc88-tdzsw          coredns                   7m           12Mi

Save your progress? Sign up for free

Drill 3: An App with CPU Requests

The example image from the Kubernetes docs performs a computation on every request:

k create ns tag06
k config set-context --current --namespace=tag06
k create deployment php --image=registry.k8s.io/hpa-example --port=80
k set resources deployment php --requests=cpu=200m --limits=cpu=500m
k expose deployment php --port=80

Expected:

deployment.apps/php created
deployment.apps/php resource requirements updated
service/php exposed

Save your progress? Sign up for free

Drill 4: Horizontal Pod Autoscaler

k autoscale deployment php --cpu=50% --min=1 --max=5
k get hpa php

Expected:

horizontalpodautoscaler.autoscaling/php autoscaled
NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
php    Deployment/php   cpu: <unknown>/50%   1         5         1          68s

<unknown> disappears after one to two minutes, as soon as metrics are available. Until then, k describe hpa php reports events such as FailedGetResourceMetric – that is normal.

The generated object is an autoscaling/v2 HPA:

k autoscale deployment php --cpu=50% --min=1 --max=5 $do | head -14

Expected:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php
spec:
  maxReplicas: 5
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 50
        type: Utilization
    type: Resource
  minReplicas: 1

Save your progress? Sign up for free

Drill 5: Generating Load

k run load --image=busybox:1.37 --restart=Never -- \
  sh -c 'while true; do wget -q -O- http://php >/dev/null; done'

After about 90 seconds:

k get hpa php
k top pods

Expected (values fluctuate):

NAME   REFERENCE        TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
php    Deployment/php   cpu: 65%/50%   1         5         5          2m45s
NAME                  CPU(cores)   MEMORY(bytes)
load                  11m          0Mi
php-6b569995f-5kmct   219m         11Mi
php-6b569995f-6gwhf   20m          11Mi
php-6b569995f-bwtjd   129m         10Mi
php-6b569995f-cfqrh   149m         12Mi
php-6b569995f-dtdsk   58m          11Mi

Five replicas – the maximum. Stop the load:

k delete pod load --now

Save your progress? Sign up for free

Drill 6: Scaling Down Faster

By default, the HPA waits five minutes before removing Pods – protection against flapping. For the lab, we shorten the window:

k patch hpa php \
  -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":30}}}}'
k get hpa php -o jsonpath='{.spec.behavior.scaleDown}{"\n"}'

Expected:

horizontalpodautoscaler.autoscaling/php patched
{"policies":[{"periodSeconds":15,"type":"Percent","value":100}],"selectPolicy":"Max",
"stabilizationWindowSeconds":30}

After a little over a minute:

k get hpa php

Expected: REPLICAS is back at 1.

Save your progress? Sign up for free

Drill 7: In-Place Resize

Since Kubernetes 1.35, the resize subresource changes the CPU and memory of a running Pod:

cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: resize
spec:
  containers:
  - name: app
    image: nginx:1.29
    resources:
      requests:
        cpu: 100m
        memory: 64Mi
      limits:
        cpu: 200m
        memory: 128Mi
EOF
k wait --for=condition=Ready pod/resize
k patch pod resize --subresource resize -p '{"spec":{"containers":[{"name":"app",
  "resources":{"requests":{"cpu":"300m"},"limits":{"cpu":"500m"}}}]}}'
k get pod resize -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}'
k get pod resize -o jsonpath='{.status.containerStatuses[0].restartCount}{"\n"}'

Expected:

pod/resize patched
{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"300m","memory":"64Mi"}}
0

The new values apply, and the container was not restarted. A normal k apply with changed resources, on the other hand, would be rejected (Day 2).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the node with the highest CPU load first. Target time: 15 s
  2. Find the Pod in the whole cluster that uses the most memory and write its name to /tmp/top-pod.txt. Target time: 60 s
  3. Create an HPA for php: 2 to 8 replicas, target 70 % CPU. Target time: 20 s

Solution

k top node --sort-by=cpu
k top pods -A --sort-by=memory --no-headers | head -1 \
  | awk '{print $2}' > /tmp/top-pod.txt
cat /tmp/top-pod.txt
k delete hpa php
k autoscale deployment php --cpu=70% --min=2 --max=8

Expected (excerpt):

kube-apiserver-cp1
horizontalpodautoscaler.autoscaling/php autoscaled

Save your progress? Sign up for free

Checkpoint

k get apiservice v1beta1.metrics.k8s.io

Expected: AVAILABLE is True.

k get hpa php -o jsonpath='{.spec.minReplicas} {.spec.maxReplicas}{"\n"}'

Expected: 2 8

k get pod resize -o jsonpath='{.spec.containers[0].resources.requests.cpu}{"\n"}'

Expected: 300m

Challenge

The memory limit of the Pod resize should go up to 256Mi – without a restart.

Hint: Memory changes go through the same subresource as CPU.

Solution

k patch pod resize --subresource resize \
  -p '{"spec":{"containers":[{"name":"app",
  "resources":{"limits":{"memory":"256Mi"}}}]}}'
k get pod resize -o jsonpath='{.status.containerStatuses[0].resources.limits}{"\n"}'
k get pod resize -o jsonpath='{.status.containerStatuses[0].restartCount}{"\n"}'

Expected:

pod/resize patched
{"cpu":"500m","memory":"256Mi"}
0

Clean up (metrics-server stays for the following days):

k config set-context --current --namespace=default
k delete ns tag06

Save your progress? Sign up for free

Exam Tip

  • k top is the fastest answer to “which Pod uses the most” – with --sort-by and -A.
  • HPA stuck at <unknown>? Either the CPU request is missing or metrics-server is not running.
  • Remember this docs page: “HorizontalPodAutoscaler Walkthrough”.

Enterprise Note

  • Instead of --kubelet-insecure-tls: set serverTLSBootstrap: true in the kubelet and approve the serving CSRs (or have them approved) – then metrics-server verifies real certificates.
  • kubectl top only shows the current moment. For history and alerts, use Prometheus (book “Observability hands-on”).
  • For event-driven scaling (queues, Kafka), use KEDA; for nodes, Cluster Autoscaler or Karpenter.