Day 1: Reading Metrics by Hand
Goal: You read the
/metricsof notes-api without tools, recognize counter, gauge and histogram, and know why each pod has its own numbers. · Duration: approx. 45 min · Prerequisite: Day 0
Briefing
- A metric is a number with a name and
labels, for example
notes_http_requests_total{method="GET",path="/notes",code="200"} 188. Every combination of name and labels is its own time series. - notes-api exposes its metrics at
/metricsas text. From Day 2 on, Prometheus fetches this page every few seconds and remembers every value with a timestamp. This fetching is called a scrape. - Three types cover almost everything: counters only count up, gauges go up and down, histograms sort measurements into buckets.
- Today you only work with
curlandgrep. That way you see exactly what Prometheus sees later.
curl ──► :30080/metrics ──► Service notes-api ──► Pod A or Pod B
curl ──► :9081 (port-forward) ──────────────────► Pod A
Drill 1: The /metrics Page
curl -s localhost:30080/metrics | grep "^# TYPE notes_"Expected:
# TYPE notes_build_info gauge
# TYPE notes_created_total counter
# TYPE notes_db_up gauge
# TYPE notes_http_request_duration_seconds histogram
# TYPE notes_http_requests_total counter
- Every metric is preceded by two comment lines:
# HELPwith a description and# TYPEwith the type. Prometheus reads both. - The prefix
notes_shows where the metric comes from. The suffix names the unit (_seconds) or says that something is being counted (_total).
In addition, the library delivers its own metrics about Go and the process:
curl -s localhost:30080/metrics | grep -c "^[a-z]"Expected (roughly):
113
grep -c counts the lines that start with a lowercase
letter – that is, all samples, no comments. About half start with
go_ or process_.
Save your progress? Sign up for free
Drill 2: Counters
curl -s localhost:30080/metrics | grep "^notes_http_requests_total"Expected (the numbers differ):
notes_http_requests_total{code="200",method="GET",path="/healthz"} 14
notes_http_requests_total{code="200",method="GET",path="/metrics"} 2
notes_http_requests_total{code="200",method="GET",path="/notes"} 292
notes_http_requests_total{code="201",method="POST",path="/notes"} 70
Repeat the command after a few seconds. The numbers for
/notes have grown.
- A counter only counts up. When the pod restarts, it starts at 0 again.
- The absolute value says little: 292 requests since when? What’s
interesting is how fast the counter grows. Prometheus calculates that on
Day 3 with
rate(). - The labels split the counter: every combination of method, path and
status gets its own line.
/healthzis the Kubernetes readiness probe,/metricswas you in Drill 1.
Save your progress? Sign up for free
Drill 3: Gauges
curl -s localhost:30080/metrics \
| grep "^\(notes_db_up\|notes_build_info\|go_goroutines\|process_resident\)"Expected:
go_goroutines 11
notes_build_info{store="postgres",version="0.8.0"} 1
notes_db_up 1
process_resident_memory_bytes 1.968128e+07
- A gauge is a point-in-time value: it goes up and
down.
go_goroutinescounts the running goroutines,process_resident_memory_bytesthe memory in use – about 20 MB here. - notes-api sets
notes_db_upto 1 or 0 on every health check. On Day 4 this becomes an alert. notes_build_infoalways has the value 1. The information is in the labels: which version is running, with which store? This pattern is called an info metric.
Save your progress? Sign up for free
Drill 4: Histograms
curl -s localhost:30080/metrics | grep 'method="GET",path="/notes"'Expected (shortened, the numbers differ):
..._bucket{method="GET",path="/notes",le="0.005"} 256
..._bucket{method="GET",path="/notes",le="0.01"} 268
..._bucket{method="GET",path="/notes",le="0.025"} 272
..._bucket{method="GET",path="/notes",le="0.05"} 273
..._bucket{method="GET",path="/notes",le="0.1"} 274
...
..._bucket{method="GET",path="/notes",le="+Inf"} 274
..._sum{method="GET",path="/notes"} 0.8796131480000006
..._count{method="GET",path="/notes"} 274
notes_http_requests_total{code="200",method="GET",path="/notes"} 274
... stands for
notes_http_request_duration_seconds.
- Every request lands in all buckets whose limit
le(“less or equal”) it doesn’t exceed. 256 requests took at most 5 ms, 268 at most 10 ms. The buckets are cumulative. le="+Inf"contains everything and is therefore equal to_count._sumis the sum of all durations in seconds._sum / _countgives the average: 0.88 s / 274 ≈ 3 ms.- The code defines the limits, from 5 ms to 2.5 s. A histogram can only be as precise as its buckets.
Save your progress? Sign up for free
Drill 5: Measuring a Fault
notes-api can slow itself down on purpose. Set a delay of 300 ms:
kubectl -n app set env deploy/notes-api NOTES_DELAY=300ms
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
set env changes the environment in the Deployment.
Kubernetes then replaces both pods, the counters start at 0. Wait half a
minute, then:
curl -s localhost:30080/metrics | grep 'bucket{method="GET",path="/notes"'Expected (the numbers differ):
..._bucket{method="GET",path="/notes",le="0.005"} 0
..._bucket{method="GET",path="/notes",le="0.01"} 0
..._bucket{method="GET",path="/notes",le="0.025"} 0
..._bucket{method="GET",path="/notes",le="0.05"} 0
..._bucket{method="GET",path="/notes",le="0.1"} 0
..._bucket{method="GET",path="/notes",le="0.25"} 0
..._bucket{method="GET",path="/notes",le="0.5"} 47
..._bucket{method="GET",path="/notes",le="1"} 47
..._bucket{method="GET",path="/notes",le="2.5"} 47
..._bucket{method="GET",path="/notes",le="+Inf"} 47
- All requests are between 250 and 500 ms. Up to the limit 0.25 it’s 0 everywhere.
- The histogram doesn’t know anything more precise than “between 0.25 and 0.5 s”. That is enough for alerts and dashboards.
Remove the delay again – the minus after the name deletes the variable:
kubectl -n app set env deploy/notes-api NOTES_DELAY-
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
Save your progress? Sign up for free
Drill 6: Every Pod Counts for Itself
Through the Service, every request lands on some pod – including
those to /metrics. With port-forward you talk
to a specific pod directly:
kubectl -n app get pods -l app.kubernetes.io/name=notes-apiExpected:
NAME READY STATUS RESTARTS AGE
notes-api-57f4f47769-s8cmj 1/1 Running 0 67s
notes-api-57f4f47769-vd8wp 1/1 Running 0 63s
If there is still a third pod with Completed or
Terminating, that’s an old one from Drill 5. Wait until
it’s gone.
Insert your pod names:
kubectl -n app port-forward notes-api-57f4f47769-s8cmj 9081:8080 >/dev/null &
kubectl -n app port-forward notes-api-57f4f47769-vd8wp 9082:8080 >/dev/null &
curl -s localhost:9081/metrics | grep ^notes_created_total
curl -s localhost:9082/metrics | grep ^notes_created_totalExpected (the numbers differ):
notes_created_total 30
notes_created_total 35
port-forwardforwards a port of the VM to a pod. The&sends the command to the background, the shell stays free.- Every pod only counts what it handled itself. The truth is the sum of both numbers.
- That’s why Prometheus never asks the Service, but each pod individually. How it knows the pods, you see on Day 2.
Stop the port forwards:
kill %1 %2Save your progress? Sign up for free
Drill 7: Where the Numbers Come From
grep -n "Name:" ~/obs-lab/code/notes-api/metrics.goExpected:
30: Name: "notes_http_requests_total",
34: Name: "notes_http_request_duration_seconds",
39: Name: "notes_created_total",
43: Name: "notes_db_up",
48: Name: "notes_build_info",
- The metrics are created with
client_golang, the official Go library of Prometheus. There are counterparts for Java, Python, .NET and Node. - A middleware at the end of the file wraps every handler, counts every request and measures its duration. The handlers themselves know nothing about metrics.
Call a path that doesn’t exist:
curl -s localhost:30080/gibtsnicht
curl -s localhost:30080/metrics | grep 'path="other"'Expected (the second line may stay empty – then the other pod answered):
404 page not found
notes_http_requests_total{code="404",method="GET",path="other"} 1
Unknown paths land under other. If notes-api stored
every path as a label, any visitor could create any number of time
series with made-up URLs. This problem is called
cardinality and comes back on Day 14.
Save your progress? Sign up for free
Checkpoint
curl -s localhost:30080/metrics | grep -c 'bucket{method="GET",path="/notes"'Expected:
10
Nine limits from the code plus +Inf.
kubectl -n app set env deploy/notes-api --listExpected:
# Deployment notes-api, container notes-api
# NOTES_API_KEY from secret notes, key api-key
# NOTES_DB_URL from secret notes, key db-url
# Deployment notes-api, container migrate
# NOTES_DB_URL from secret notes, key db-url
--list shows the environment without changing anything.
NOTES_DELAY is gone again, no fault active anymore.
Challenge
Make a time series with code="401" show up in
/metrics.
Hint: Which request does notes-api reject with 401? And remember Drill 6.
Solution
POST /notes without an API key:
curl -s -d '{"text":"x"}' localhost:30080/notes
curl -s localhost:30080/metrics | grep 'code="401"'Expected:
{"error":"invalid api key"}
notes_http_requests_total{code="401",method="POST",path="/notes"} 1
If the second line stays empty, the request went to one pod and the
/metrics query to the other. Repeat both commands a few
times.
Save your progress? Sign up for free
Enterprise Note
- Names by convention: Base units (seconds, bytes),
_totalfor counters, one prefix per application. The rules are atprometheus.io/docs/practices/naming. If you follow them, you get ready-made dashboards and alerts from others without renaming. - No IDs as labels: User IDs, note IDs or full URLs never belong in labels. Every new value is a new time series – and memory in Prometheus.
- OpenMetrics:
/metricsalso speaks the standardized OpenMetrics format when the scraper asks for it. Prometheus does, and that’s how it gets the exemplars from Day 11.