Skip to content
Hands-on IT IT Drills

Day 1: Reading Metrics by Hand

Goal: You read the /metrics of notes-api without tools, recognize counter, gauge and histogram, and know why each pod has its own numbers. · Duration: approx. 45 min · Prerequisite: Day 0

Briefing

  • A metric is a number with a name and labels, for example notes_http_requests_total{method="GET",path="/notes",code="200"} 188. Every combination of name and labels is its own time series.
  • notes-api exposes its metrics at /metrics as text. From Day 2 on, Prometheus fetches this page every few seconds and remembers every value with a timestamp. This fetching is called a scrape.
  • Three types cover almost everything: counters only count up, gauges go up and down, histograms sort measurements into buckets.
  • Today you only work with curl and grep. That way you see exactly what Prometheus sees later.
 curl ──► :30080/metrics ──► Service notes-api ──► Pod A  or  Pod B
 curl ──► :9081 (port-forward) ──────────────────► Pod A

Drill 1: The /metrics Page

curl -s localhost:30080/metrics | grep "^# TYPE notes_"

Expected:

# TYPE notes_build_info gauge
# TYPE notes_created_total counter
# TYPE notes_db_up gauge
# TYPE notes_http_request_duration_seconds histogram
# TYPE notes_http_requests_total counter
  • Every metric is preceded by two comment lines: # HELP with a description and # TYPE with the type. Prometheus reads both.
  • The prefix notes_ shows where the metric comes from. The suffix names the unit (_seconds) or says that something is being counted (_total).

In addition, the library delivers its own metrics about Go and the process:

curl -s localhost:30080/metrics | grep -c "^[a-z]"

Expected (roughly):

113

grep -c counts the lines that start with a lowercase letter – that is, all samples, no comments. About half start with go_ or process_.

Save your progress? Sign up for free

Drill 2: Counters

curl -s localhost:30080/metrics | grep "^notes_http_requests_total"

Expected (the numbers differ):

notes_http_requests_total{code="200",method="GET",path="/healthz"} 14
notes_http_requests_total{code="200",method="GET",path="/metrics"} 2
notes_http_requests_total{code="200",method="GET",path="/notes"} 292
notes_http_requests_total{code="201",method="POST",path="/notes"} 70

Repeat the command after a few seconds. The numbers for /notes have grown.

  • A counter only counts up. When the pod restarts, it starts at 0 again.
  • The absolute value says little: 292 requests since when? What’s interesting is how fast the counter grows. Prometheus calculates that on Day 3 with rate().
  • The labels split the counter: every combination of method, path and status gets its own line. /healthz is the Kubernetes readiness probe, /metrics was you in Drill 1.

Save your progress? Sign up for free

Drill 3: Gauges

curl -s localhost:30080/metrics \
  | grep "^\(notes_db_up\|notes_build_info\|go_goroutines\|process_resident\)"

Expected:

go_goroutines 11
notes_build_info{store="postgres",version="0.8.0"} 1
notes_db_up 1
process_resident_memory_bytes 1.968128e+07
  • A gauge is a point-in-time value: it goes up and down. go_goroutines counts the running goroutines, process_resident_memory_bytes the memory in use – about 20 MB here.
  • notes-api sets notes_db_up to 1 or 0 on every health check. On Day 4 this becomes an alert.
  • notes_build_info always has the value 1. The information is in the labels: which version is running, with which store? This pattern is called an info metric.

Save your progress? Sign up for free

Drill 4: Histograms

curl -s localhost:30080/metrics | grep 'method="GET",path="/notes"'

Expected (shortened, the numbers differ):

..._bucket{method="GET",path="/notes",le="0.005"} 256
..._bucket{method="GET",path="/notes",le="0.01"} 268
..._bucket{method="GET",path="/notes",le="0.025"} 272
..._bucket{method="GET",path="/notes",le="0.05"} 273
..._bucket{method="GET",path="/notes",le="0.1"} 274
...
..._bucket{method="GET",path="/notes",le="+Inf"} 274
..._sum{method="GET",path="/notes"} 0.8796131480000006
..._count{method="GET",path="/notes"} 274
notes_http_requests_total{code="200",method="GET",path="/notes"} 274

... stands for notes_http_request_duration_seconds.

  • Every request lands in all buckets whose limit le (“less or equal”) it doesn’t exceed. 256 requests took at most 5 ms, 268 at most 10 ms. The buckets are cumulative.
  • le="+Inf" contains everything and is therefore equal to _count.
  • _sum is the sum of all durations in seconds. _sum / _count gives the average: 0.88 s / 274 ≈ 3 ms.
  • The code defines the limits, from 5 ms to 2.5 s. A histogram can only be as precise as its buckets.

Save your progress? Sign up for free

Drill 5: Measuring a Fault

notes-api can slow itself down on purpose. Set a delay of 300 ms:

kubectl -n app set env deploy/notes-api NOTES_DELAY=300ms
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

set env changes the environment in the Deployment. Kubernetes then replaces both pods, the counters start at 0. Wait half a minute, then:

curl -s localhost:30080/metrics | grep 'bucket{method="GET",path="/notes"'

Expected (the numbers differ):

..._bucket{method="GET",path="/notes",le="0.005"} 0
..._bucket{method="GET",path="/notes",le="0.01"} 0
..._bucket{method="GET",path="/notes",le="0.025"} 0
..._bucket{method="GET",path="/notes",le="0.05"} 0
..._bucket{method="GET",path="/notes",le="0.1"} 0
..._bucket{method="GET",path="/notes",le="0.25"} 0
..._bucket{method="GET",path="/notes",le="0.5"} 47
..._bucket{method="GET",path="/notes",le="1"} 47
..._bucket{method="GET",path="/notes",le="2.5"} 47
..._bucket{method="GET",path="/notes",le="+Inf"} 47
  • All requests are between 250 and 500 ms. Up to the limit 0.25 it’s 0 everywhere.
  • The histogram doesn’t know anything more precise than “between 0.25 and 0.5 s”. That is enough for alerts and dashboards.

Remove the delay again – the minus after the name deletes the variable:

kubectl -n app set env deploy/notes-api NOTES_DELAY-
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

Save your progress? Sign up for free

Drill 6: Every Pod Counts for Itself

Through the Service, every request lands on some pod – including those to /metrics. With port-forward you talk to a specific pod directly:

kubectl -n app get pods -l app.kubernetes.io/name=notes-api

Expected:

NAME                         READY   STATUS    RESTARTS   AGE
notes-api-57f4f47769-s8cmj   1/1     Running   0          67s
notes-api-57f4f47769-vd8wp   1/1     Running   0          63s

If there is still a third pod with Completed or Terminating, that’s an old one from Drill 5. Wait until it’s gone.

Insert your pod names:

kubectl -n app port-forward notes-api-57f4f47769-s8cmj 9081:8080 >/dev/null &
kubectl -n app port-forward notes-api-57f4f47769-vd8wp 9082:8080 >/dev/null &
curl -s localhost:9081/metrics | grep ^notes_created_total
curl -s localhost:9082/metrics | grep ^notes_created_total

Expected (the numbers differ):

notes_created_total 30
notes_created_total 35
  • port-forward forwards a port of the VM to a pod. The & sends the command to the background, the shell stays free.
  • Every pod only counts what it handled itself. The truth is the sum of both numbers.
  • That’s why Prometheus never asks the Service, but each pod individually. How it knows the pods, you see on Day 2.

Stop the port forwards:

kill %1 %2

Save your progress? Sign up for free

Drill 7: Where the Numbers Come From

grep -n "Name:" ~/obs-lab/code/notes-api/metrics.go

Expected:

30:         Name: "notes_http_requests_total",
34:         Name:    "notes_http_request_duration_seconds",
39:         Name: "notes_created_total",
43:         Name: "notes_db_up",
48:     Name:        "notes_build_info",
  • The metrics are created with client_golang, the official Go library of Prometheus. There are counterparts for Java, Python, .NET and Node.
  • A middleware at the end of the file wraps every handler, counts every request and measures its duration. The handlers themselves know nothing about metrics.

Call a path that doesn’t exist:

curl -s localhost:30080/gibtsnicht
curl -s localhost:30080/metrics | grep 'path="other"'

Expected (the second line may stay empty – then the other pod answered):

404 page not found
notes_http_requests_total{code="404",method="GET",path="other"} 1

Unknown paths land under other. If notes-api stored every path as a label, any visitor could create any number of time series with made-up URLs. This problem is called cardinality and comes back on Day 14.

Save your progress? Sign up for free

Checkpoint

curl -s localhost:30080/metrics | grep -c 'bucket{method="GET",path="/notes"'

Expected:

10

Nine limits from the code plus +Inf.

kubectl -n app set env deploy/notes-api --list

Expected:

# Deployment notes-api, container notes-api
# NOTES_API_KEY from secret notes, key api-key
# NOTES_DB_URL from secret notes, key db-url
# Deployment notes-api, container migrate
# NOTES_DB_URL from secret notes, key db-url

--list shows the environment without changing anything. NOTES_DELAY is gone again, no fault active anymore.

Challenge

Make a time series with code="401" show up in /metrics.

Hint: Which request does notes-api reject with 401? And remember Drill 6.

Solution

POST /notes without an API key:

curl -s -d '{"text":"x"}' localhost:30080/notes
curl -s localhost:30080/metrics | grep 'code="401"'

Expected:

{"error":"invalid api key"}
notes_http_requests_total{code="401",method="POST",path="/notes"} 1

If the second line stays empty, the request went to one pod and the /metrics query to the other. Repeat both commands a few times.

Save your progress? Sign up for free

Enterprise Note

  • Names by convention: Base units (seconds, bytes), _total for counters, one prefix per application. The rules are at prometheus.io/docs/practices/naming. If you follow them, you get ready-made dashboards and alerts from others without renaming.
  • No IDs as labels: User IDs, note IDs or full URLs never belong in labels. Every new value is a new time series – and memory in Prometheus.
  • OpenMetrics: /metrics also speaks the standardized OpenMetrics format when the scraper asks for it. Prometheus does, and that’s how it gets the exemplars from Day 11.