Skip to content
Hands-on IT IT Drills

Day 13: Observing Kubernetes

Goal: You read memory, CPU and restarts of the pods from Prometheus, recognize an OOMKill in metrics and events, and know which ready-made alerts the chart brings for it. · Duration: approx. 60 min · Prerequisite: Day 12

Briefing

  • So far you have observed notes-api from the inside: with metrics, logs and traces that the application delivers itself. Kubernetes sees it from the outside – and that is often enough to find a problem before you know the code.
  • Three sources deliver the picture, all in the cluster since Day 2:
    • cAdvisor in the kubelet measures what containers consume: CPU, memory, network.
    • kube-state-metrics translates what Kubernetes knows: limits, restarts, desired and ready replicas, reasons why a container ended.
    • node-exporter measures the machine underneath.
  • Events are the fourth source: short messages like “Back-off restarting failed container”. Kubernetes forgets them after an hour – unless someone collects them.
 Kubelet/cAdvisor ─► container_*        (consumption)
 kube-state-metrics ─► kube_*           (state)        ──► Prometheus
 node-exporter ─► node_*                (machine)
 Events ─► Alloy ─► Loki                (messages)

Drill 1: Memory and CPU

promtool query instant http://localhost:30090 \
  'sum by (pod) (container_memory_working_set_bytes{namespace="app", container!=""})'

Expected (the numbers differ):

{pod="last-6fdd4b976d-47f97"} => 3448832 @[...]
{pod="postgres-0"} => 64946176 @[...]
{pod="notes-api-76d9df4449-zgs5d"} => 8589312 @[...]
{pod="notes-api-76d9df4449-8rszn"} => 8118272 @[...]
  • working_set is the memory the kernel can’t simply free. Kubernetes measures the limit against this number – if it exceeds it, the OOMKill follows.
  • container!="" throws away the time series for the whole pod that cAdvisor also delivers. Without the filter, everything counts twice.
promtool query instant http://localhost:30090 \
  'sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="app",
     container!=""}[5m]))'

Expected (shortened):

{pod="last-6fdd4b976d-47f97"} => 0.014055368674208252 @[...]
{pod="postgres-0"} => 0.046181930503317176 @[...]
{pod="notes-api-76d9df4449-zgs5d"} => 0.015034315387111894 @[...]
{pod="notes-api-76d9df4449-8rszn"} => 0.014257365085933305 @[...]
  • CPU is a counter in seconds. rate() turns it into “cores”: 0.046 means 46 millicores, so 4.6 percent of a core.
  • Pods that no longer exist stay in the window for five more minutes – as on Day 3.

Save your progress? Sign up for free

Drill 2: Consumption Against the Limit

How close is each container to its memory limit?

promtool query instant http://localhost:30090 '
  max by (pod, container) (container_memory_working_set_bytes{namespace="app",
      container!=""})
    / max by (pod, container) (kube_pod_container_resource_limits{namespace="app",
        resource="memory"})'

Expected (shortened):

{container="lesen", pod="last-6fdd4b976d-47f97"} => 0.0633544921875 @[...]
{container="postgres", pod="postgres-0"} => 0.249237060546875 @[...]
{container="schreiben", pod="last-6fdd4b976d-47f97"} => 0.0465087890625 @[...]
{container="notes-api", pod="notes-api-76d9df4449-zgs5d"} => 0.063995361328125 @[...]
{container="notes-api", pod="notes-api-76d9df4449-8rszn"} => 0.06048583984375 @[...]
  • The division combines two metrics from different sources. PromQL pairs time series with the same labels – that’s why max by (pod, container) reduces both sides to the same two labels.
  • Postgres uses a quarter of its 256 Mi. notes-api is generously equipped with 128 Mi.

Save your progress? Sign up for free

Drill 3: A Container That Wants Too Much

A pod that eats memory until the limit strikes:

cd ~/obs-lab
cat > speicherfresser.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: speicherfresser
  namespace: app
  labels:
    app.kubernetes.io/name: speicherfresser
spec:
  containers:
    - name: fresser
      image: busybox:1.37
      command: [sh, -c, "sleep 20; tail /dev/zero"]
      resources:
        requests: {memory: 16Mi}
        limits: {memory: 32Mi}
EOF
kubectl apply -f speicherfresser.yaml

Expected:

pod/speicherfresser created

tail /dev/zero searches an endless stream of null bytes for the end of the line and keeps everything in memory. After 20 seconds of sleep, it takes a moment to reach the 32 Mi. After a minute:

kubectl -n app get pod speicherfresser

Expected (depending on the moment Running, OOMKilled or CrashLoopBackOff):

NAME              READY   STATUS    RESTARTS      AGE
speicherfresser   1/1     Running   2 (16s ago)   61s

Kubernetes keeps restarting the container and waits longer and longer in between. What does Prometheus say?

promtool query instant http://localhost:30090 \
  'kube_pod_container_status_last_terminated_reason{namespace="app"} == 1'

Expected (shortened):

kube_pod_container_status_last_terminated_reason{container="fresser", ...,
pod="speicherfresser", reason="OOMKilled", ...} => 1 @[...]
promtool query instant http://localhost:30090 \
  'increase(kube_pod_container_status_restarts_total{pod="speicherfresser"}[5m])'

Expected (roughly):

{container="fresser", ...} => 1.6 @[...]

Not a whole number: increase extrapolates to the edges of the window, as on Day 3.

  • kube-state-metrics knows the reason for the last termination: OOMKilled, Error, Completed. As a label with value 1 – an info pattern like notes_build_info.
  • Restarts are a counter. increase over five minutes is the basis of the ready-made alert KubePodCrashLooping.

Save your progress? Sign up for free

Drill 4: The Ready-Made Alerts

promtool query instant http://localhost:30090 'ALERTS{namespace="app"}'

Expected (after two to three minutes, shortened):

ALERTS{alertname="NotesSloSlowBurn", alertstate="firing", namespace="app",
severity="warning"} => 1 @[...]
ALERTS{alertname="KubePodNotReady", alertstate="pending", ..., pod="speicherfresser",
severity="warning"} => 1 @[...]
  • NotesSloSlowBurn is your alert from Day 12: the total outage there is still in the six-hour window – and, if you continued right away, also still in the 30-minute window. If the line is missing, Day 12 was a while ago.
  • KubePodNotReady reports the memory eater. Depending on the moment, you may also see KubePodCrashLooping – both are pending.

You didn’t write the Kubernetes alerts. They come from the chart’s rules:

kubectl -n monitoring get prometheusrule -o name | wc -l
kubectl -n monitoring get prometheusrule kps-kubernetes-apps \
  -o jsonpath='{.spec.groups[0].rules[0].expr}{"\n"}'

Expected:

30
max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff", job...
  • 30 PrometheusRule objects with rules for nodes, pods, volumes, the API server and the stack itself. They come from the kubernetes-mixin project.
  • The second line is the rule of KubePodCrashLooping. It fires after 15 minutes (for: 15m); until then it stays pending. You don’t have to wait – the mechanism is the same as on Day 4.

Save your progress? Sign up for free

Drill 5: Collecting Events

What Kubernetes has to say about the memory eater:

kubectl -n app get events --field-selector involvedObject.name=speicherfresser

Expected (shortened):

LAST SEEN   TYPE      REASON      OBJECT                MESSAGE
2m33s       Normal    Scheduled   pod/speicherfresser   Successfully assigned app/spe...
2m32s       Normal    Pulling     pod/speicherfresser   Pulling image "busybox:1.37"
...
32s         Warning   BackOff     pod/speicherfresser   Back-off restarting failed co...

These messages are gone after an hour. Alloy can send them to Loki – one more component in the chain. Here is the whole alloy.yaml:

cat > alloy.yaml <<'EOF'
controller:
  type: deployment

alloy:
  resources:
    requests: {cpu: 20m, memory: 64Mi}
    limits: {memory: 256Mi}
  configMap:
    content: |
      discovery.kubernetes "pods" {
        role = "pod"
      }

      discovery.relabel "pods" {
        targets = discovery.kubernetes.pods.targets
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          target_label  = "app"
        }
      }

      loki.source.kubernetes "pods" {
        targets    = discovery.relabel.pods.output
        forward_to = [loki.write.default.receiver]
      }

      loki.source.kubernetes_events "events" {
        job_name   = "kubernetes-events"
        log_format = "logfmt"
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki:3100/loki/api/v1/push"
        }
      }
EOF
helm upgrade alloy grafana/alloy --version 1.15.0 -n monitoring -f alloy.yaml --wait

Expected (shortened):

Release "alloy" has been upgraded. Happy Helming!
...
STATUS: deployed

loki.source.kubernetes_events watches the events of all namespaces and writes each one as a log line – with job="kubernetes-events" and the namespace as a label.

After another restart of the memory eater:

logcli query --quiet -o raw --limit 3 --since 5m \
  '{job="kubernetes-events", namespace="app"} |= "speicherfresser"'

Expected (shortened):

name=speicherfresser kind=Pod ... reason=Started type=Normal count=6 msg="Contai...
name=speicherfresser kind=Pod ... reason=Created type=Normal count=6 msg="Contai...
name=speicherfresser kind=Pod ... reason=Pulled type=Normal count=5 msg="Contain...

count=6: Kubernetes combines identical events and keeps count.

Now the events stay for three days – as long as all logs.

Save your progress? Sign up for free

Checkpoint

Clean up the memory eater:

kubectl -n app delete pod speicherfresser

Expected:

pod "speicherfresser" deleted from app namespace
logcli query --quiet -o raw --limit 1 --since 15m \
  '{job="kubernetes-events", namespace="app"} |= "Killing"'

Expected (shortened):

name=speicherfresser kind=Pod ... reason=Killing type=Normal count=1 msg="Stopping co...

Deleting is an event too – and now lands in Loki.

Challenge

Which three containers in the whole cluster are using the most memory right now?

Hint: topk(3, …) keeps the three largest time series. No filter on the namespace.

Solution

promtool query instant http://localhost:30090 \
  'topk(3, max by (namespace, container) (
     container_memory_working_set_bytes{container!=""}))'

Expected (roughly):

{container="prometheus", namespace="monitoring"} => 530051072 @[...]
{container="grafana", namespace="monitoring"} => 295194624 @[...]
{container="tempo", namespace="monitoring"} => 165629952 @[...]

The tools for observing need more memory than what they observe. That’s why Day 14 is about running the stack itself.

Save your progress? Sign up for free

Enterprise Note

  • Set requests honestly: The query from Drill 2 with requests instead of limits shows who reserves too much. Inflated requests waste nodes, too-low ones lead to eviction. Tools like the Vertical Pod Autoscaler suggest values.
  • CPU limits and throttling: container_cpu_cfs_throttled_periods_total shows how often a container hit its CPU limit. That’s why many teams set only CPU requests and no limits.
  • Events with care: In large clusters, events are noisy. Filter in Alloy by type=Warning or by namespaces before they land in Loki.