Day 13: Observing Kubernetes
Goal: You read memory, CPU and restarts of the pods from Prometheus, recognize an OOMKill in metrics and events, and know which ready-made alerts the chart brings for it. · Duration: approx. 60 min · Prerequisite: Day 12
Briefing
- So far you have observed notes-api from the inside: with metrics, logs and traces that the application delivers itself. Kubernetes sees it from the outside – and that is often enough to find a problem before you know the code.
- Three sources deliver the picture, all in the cluster since Day 2:
- cAdvisor in the kubelet measures what containers consume: CPU, memory, network.
- kube-state-metrics translates what Kubernetes knows: limits, restarts, desired and ready replicas, reasons why a container ended.
- node-exporter measures the machine underneath.
- Events are the fourth source: short messages like “Back-off restarting failed container”. Kubernetes forgets them after an hour – unless someone collects them.
Kubelet/cAdvisor ─► container_* (consumption)
kube-state-metrics ─► kube_* (state) ──► Prometheus
node-exporter ─► node_* (machine)
Events ─► Alloy ─► Loki (messages)
Drill 1: Memory and CPU
promtool query instant http://localhost:30090 \
'sum by (pod) (container_memory_working_set_bytes{namespace="app", container!=""})'Expected (the numbers differ):
{pod="last-6fdd4b976d-47f97"} => 3448832 @[...]
{pod="postgres-0"} => 64946176 @[...]
{pod="notes-api-76d9df4449-zgs5d"} => 8589312 @[...]
{pod="notes-api-76d9df4449-8rszn"} => 8118272 @[...]
working_setis the memory the kernel can’t simply free. Kubernetes measures the limit against this number – if it exceeds it, the OOMKill follows.container!=""throws away the time series for the whole pod that cAdvisor also delivers. Without the filter, everything counts twice.
promtool query instant http://localhost:30090 \
'sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="app",
container!=""}[5m]))'Expected (shortened):
{pod="last-6fdd4b976d-47f97"} => 0.014055368674208252 @[...]
{pod="postgres-0"} => 0.046181930503317176 @[...]
{pod="notes-api-76d9df4449-zgs5d"} => 0.015034315387111894 @[...]
{pod="notes-api-76d9df4449-8rszn"} => 0.014257365085933305 @[...]
- CPU is a counter in seconds.
rate()turns it into “cores”: 0.046 means 46 millicores, so 4.6 percent of a core. - Pods that no longer exist stay in the window for five more minutes – as on Day 3.
Save your progress? Sign up for free
Drill 2: Consumption Against the Limit
How close is each container to its memory limit?
promtool query instant http://localhost:30090 '
max by (pod, container) (container_memory_working_set_bytes{namespace="app",
container!=""})
/ max by (pod, container) (kube_pod_container_resource_limits{namespace="app",
resource="memory"})'Expected (shortened):
{container="lesen", pod="last-6fdd4b976d-47f97"} => 0.0633544921875 @[...]
{container="postgres", pod="postgres-0"} => 0.249237060546875 @[...]
{container="schreiben", pod="last-6fdd4b976d-47f97"} => 0.0465087890625 @[...]
{container="notes-api", pod="notes-api-76d9df4449-zgs5d"} => 0.063995361328125 @[...]
{container="notes-api", pod="notes-api-76d9df4449-8rszn"} => 0.06048583984375 @[...]
- The division combines two metrics from different sources. PromQL
pairs time series with the same labels – that’s why
max by (pod, container)reduces both sides to the same two labels. - Postgres uses a quarter of its 256 Mi. notes-api is generously equipped with 128 Mi.
Save your progress? Sign up for free
Drill 3: A Container That Wants Too Much
A pod that eats memory until the limit strikes:
cd ~/obs-lab
cat > speicherfresser.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: speicherfresser
namespace: app
labels:
app.kubernetes.io/name: speicherfresser
spec:
containers:
- name: fresser
image: busybox:1.37
command: [sh, -c, "sleep 20; tail /dev/zero"]
resources:
requests: {memory: 16Mi}
limits: {memory: 32Mi}
EOF
kubectl apply -f speicherfresser.yamlExpected:
pod/speicherfresser created
tail /dev/zero searches an endless stream of null bytes
for the end of the line and keeps everything in memory. After 20 seconds
of sleep, it takes a moment to reach the 32 Mi. After a minute:
kubectl -n app get pod speicherfresserExpected (depending on the moment Running,
OOMKilled or CrashLoopBackOff):
NAME READY STATUS RESTARTS AGE
speicherfresser 1/1 Running 2 (16s ago) 61s
Kubernetes keeps restarting the container and waits longer and longer in between. What does Prometheus say?
promtool query instant http://localhost:30090 \
'kube_pod_container_status_last_terminated_reason{namespace="app"} == 1'Expected (shortened):
kube_pod_container_status_last_terminated_reason{container="fresser", ...,
pod="speicherfresser", reason="OOMKilled", ...} => 1 @[...]
promtool query instant http://localhost:30090 \
'increase(kube_pod_container_status_restarts_total{pod="speicherfresser"}[5m])'Expected (roughly):
{container="fresser", ...} => 1.6 @[...]
Not a whole number: increase extrapolates to the edges
of the window, as on Day 3.
- kube-state-metrics knows the reason for the last termination:
OOMKilled,Error,Completed. As a label with value 1 – an info pattern likenotes_build_info. - Restarts are a counter.
increaseover five minutes is the basis of the ready-made alertKubePodCrashLooping.
Save your progress? Sign up for free
Drill 4: The Ready-Made Alerts
promtool query instant http://localhost:30090 'ALERTS{namespace="app"}'Expected (after two to three minutes, shortened):
ALERTS{alertname="NotesSloSlowBurn", alertstate="firing", namespace="app",
severity="warning"} => 1 @[...]
ALERTS{alertname="KubePodNotReady", alertstate="pending", ..., pod="speicherfresser",
severity="warning"} => 1 @[...]
- NotesSloSlowBurn is your alert from Day 12: the total outage there is still in the six-hour window – and, if you continued right away, also still in the 30-minute window. If the line is missing, Day 12 was a while ago.
- KubePodNotReady reports the memory eater. Depending on the moment, you may also see KubePodCrashLooping – both are pending.
You didn’t write the Kubernetes alerts. They come from the chart’s rules:
kubectl -n monitoring get prometheusrule -o name | wc -l
kubectl -n monitoring get prometheusrule kps-kubernetes-apps \
-o jsonpath='{.spec.groups[0].rules[0].expr}{"\n"}'Expected:
30
max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff", job...
- 30 PrometheusRule objects with rules for nodes, pods, volumes, the API server and the stack itself. They come from the kubernetes-mixin project.
- The second line is the rule of
KubePodCrashLooping. It fires after 15 minutes (for: 15m); until then it stays pending. You don’t have to wait – the mechanism is the same as on Day 4.
Save your progress? Sign up for free
Drill 5: Collecting Events
What Kubernetes has to say about the memory eater:
kubectl -n app get events --field-selector involvedObject.name=speicherfresserExpected (shortened):
LAST SEEN TYPE REASON OBJECT MESSAGE
2m33s Normal Scheduled pod/speicherfresser Successfully assigned app/spe...
2m32s Normal Pulling pod/speicherfresser Pulling image "busybox:1.37"
...
32s Warning BackOff pod/speicherfresser Back-off restarting failed co...
These messages are gone after an hour. Alloy can send them to Loki –
one more component in the chain. Here is the whole
alloy.yaml:
cat > alloy.yaml <<'EOF'
controller:
type: deployment
alloy:
resources:
requests: {cpu: 20m, memory: 64Mi}
limits: {memory: 256Mi}
configMap:
content: |
discovery.kubernetes "pods" {
role = "pod"
}
discovery.relabel "pods" {
targets = discovery.kubernetes.pods.targets
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
target_label = "app"
}
}
loki.source.kubernetes "pods" {
targets = discovery.relabel.pods.output
forward_to = [loki.write.default.receiver]
}
loki.source.kubernetes_events "events" {
job_name = "kubernetes-events"
log_format = "logfmt"
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}
EOF
helm upgrade alloy grafana/alloy --version 1.15.0 -n monitoring -f alloy.yaml --waitExpected (shortened):
Release "alloy" has been upgraded. Happy Helming!
...
STATUS: deployed
loki.source.kubernetes_events watches the events of all
namespaces and writes each one as a log line – with
job="kubernetes-events" and the namespace as a label.
After another restart of the memory eater:
logcli query --quiet -o raw --limit 3 --since 5m \
'{job="kubernetes-events", namespace="app"} |= "speicherfresser"'Expected (shortened):
name=speicherfresser kind=Pod ... reason=Started type=Normal count=6 msg="Contai...
name=speicherfresser kind=Pod ... reason=Created type=Normal count=6 msg="Contai...
name=speicherfresser kind=Pod ... reason=Pulled type=Normal count=5 msg="Contain...
count=6: Kubernetes combines identical events and keeps
count.
Now the events stay for three days – as long as all logs.
Save your progress? Sign up for free
Checkpoint
Clean up the memory eater:
kubectl -n app delete pod speicherfresserExpected:
pod "speicherfresser" deleted from app namespace
logcli query --quiet -o raw --limit 1 --since 15m \
'{job="kubernetes-events", namespace="app"} |= "Killing"'Expected (shortened):
name=speicherfresser kind=Pod ... reason=Killing type=Normal count=1 msg="Stopping co...
Deleting is an event too – and now lands in Loki.
Challenge
Which three containers in the whole cluster are using the most memory right now?
Hint: topk(3, …) keeps the three
largest time series. No filter on the namespace.
Solution
promtool query instant http://localhost:30090 \
'topk(3, max by (namespace, container) (
container_memory_working_set_bytes{container!=""}))'Expected (roughly):
{container="prometheus", namespace="monitoring"} => 530051072 @[...]
{container="grafana", namespace="monitoring"} => 295194624 @[...]
{container="tempo", namespace="monitoring"} => 165629952 @[...]
The tools for observing need more memory than what they observe. That’s why Day 14 is about running the stack itself.
Save your progress? Sign up for free
Enterprise Note
- Set requests honestly: The query from Drill 2 with
requestsinstead oflimitsshows who reserves too much. Inflated requests waste nodes, too-low ones lead to eviction. Tools like the Vertical Pod Autoscaler suggest values. - CPU limits and throttling:
container_cpu_cfs_throttled_periods_totalshows how often a container hit its CPU limit. That’s why many teams set only CPU requests and no limits. - Events with care: In large clusters, events are
noisy. Filter in Alloy by
type=Warningor by namespaces before they land in Loki.