Day 3: PromQL and Recording Rules
Goal: You calculate rates from counters and percentiles from histograms, and store frequent queries as recording rules. · Duration: approx. 60 min · Prerequisite: Day 2
Briefing
- PromQL is the query language of Prometheus. A name
like
notes_created_totalreturns an instant vector: the newest value per time series. - With
[1m]after it, it becomes a range vector: all values of the last minute per time series. Functions likerate()turn that back into one number per time series. - Aggregations like
sum by (method)combine time series and keep only the named labels. - A recording rule evaluates a query regularly and stores the result as a new time series. Dashboards and alerts then read the finished result.
notes_http_requests_total (counter, 4 time series per pod)
│ rate(...[5m]) requests per second
│ sum by (method) across all pods and codes
▼
{method="GET"} 4 {method="POST"} 1
Drill 1: Range Vector
promtool query instant http://localhost:30090 'notes_created_total[1m]'Expected (shortened):
notes_created_total{..., pod="notes-api-57f4f47769-59ks2", ...} =>
247 @[1793866895.847]
257 @[1793866910.845]
263 @[1793866925.845]
271 @[1793866940.845]
...
- Four values per pod, 15 seconds apart – the
intervalfrom the ServiceMonitor. - A range vector is raw material. Prometheus can’t draw it as a curve; you need a function that turns it into a number.
Save your progress? Sign up for free
Drill 2: rate()
promtool query instant http://localhost:30090 'rate(notes_created_total[5m])'Expected (shortened):
{..., pod="notes-api-57f4f47769-59ks2", ...} => 0.45614035087719296 @[...]
{..., pod="notes-api-57f4f47769-m66nj", ...} => 0.43859649122807015 @[...]
{..., pod="notes-api-57f4f47769-gg2rr", ...} => 0.09 @[...]
rate()takes the values of the last five minutes and calculates how much the counter grew per second. Each pod creates just under half a note per second.- The third pod is the one from the Day 2 challenge. It’s already gone but still in the window of the last five minutes. After five minutes it disappears.
- When a pod restarts, its counter drops to 0.
rate()detects that and keeps calculating correctly. That’s whyrate()belongs on every counter – never the bare number. - The window should cover at least four scrapes. At 15 seconds,
[1m]is the lower limit,[5m]smooths better.
The metric name is missing in the output: after a calculation it is
no longer a notes_created_total.
Save your progress? Sign up for free
Drill 3: Aggregating
promtool query instant http://localhost:30090 \
'sum by (method) (rate(notes_http_requests_total{path="/notes"}[5m]))'Expected:
{method="GET"} => 3.9719298245614034 @[...]
{method="POST"} => 0.983625730994152 @[...]
Four reads and one note per second – exactly what the load generator sends.
- The order matters: first
rate()per time series, thensum. The other way around, the restarts of individual pods would hide in the sum. by (method)keeps only the labelmethod.without (pod)would be the counterpart: keep all labels exceptpod.{path="/notes"}is a selector.=~allows regular expressions, such ascode=~"5.."for all server errors.
Save your progress? Sign up for free
Drill 4: increase()
promtool query instant http://localhost:30090 \
'sum(increase(notes_http_requests_total{method="POST"}[5m]))'Expected (roughly):
{} => 302.1052631578947 @[...]
increase() is rate() times the window
length: about 300 new notes in five minutes. That the result isn’t a
round number is due to the extrapolation to the edge of the window.
Save your progress? Sign up for free
Drill 5: Percentiles
An average hides outliers. How long do the slowest five percent wait?
promtool query instant http://localhost:30090 \
'histogram_quantile(0.95, sum by (le) (rate(
notes_http_request_duration_seconds_bucket{path="/notes"}[5m])))'Expected (roughly):
{} => 0.009103579487633288 @[...]
histogram_quantile(0.95, …)looks for the bucket in which the 95 percent mark lies and estimates linearly within it. 95 percent of requests are faster than about 9 ms.sum by (le)is mandatory: the labellewith the bucket limits must be preserved, everything else is combined.
Slow down notes-api as on Day 1:
kubectl -n app set env deploy/notes-api NOTES_DELAY=300ms
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
Wait five minutes – that’s how long the window is –, then the same query again:
promtool query instant http://localhost:30090 \
'histogram_quantile(0.95, sum by (le) (rate(
notes_http_request_duration_seconds_bucket{path="/notes"}[5m])))'Expected (roughly):
{} => 0.4875 @[...]
Every request takes a little over 300 ms, the result says almost 490
ms. All values are in the bucket from 0.25 to 0.5 s, and within it
histogram_quantile pretends they are evenly distributed.
Percentiles from histograms are estimates. How good they are depends on
the bucket limits.
Remove the delay:
kubectl -n app set env deploy/notes-api NOTES_DELAY-
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
Save your progress? Sign up for free
Drill 6: Recording Rules
The query from Drill 5 is long, and every dashboard would recalculate it on every load. A recording rule calculates it once every 30 seconds:
cd ~/obs-lab
cat > notes-rules.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: notes-api-records
namespace: app
spec:
groups:
- name: notes-api.records
interval: 30s
rules:
- record: job:notes_http_requests:rate5m
expr: sum by (job, method, code) (rate(notes_http_requests_total[5m]))
- record: job:notes_http_request_duration_seconds:p95_5m
expr: |
histogram_quantile(0.95, sum by (job, le) (
rate(notes_http_request_duration_seconds_bucket{path="/notes"}[5m])))
EOF
kubectl apply -f notes-rules.yamlExpected:
prometheusrule.monitoring.coreos.com/notes-api-records created
- The name follows the convention
level:metric:operation.job:says that it is aggregated across all pods of a job.:rate5mdescribes the calculation. by (job, …)keepsjobon purpose. That way it stays clear where the number comes from when more applications get the same rule later.- As with the ServiceMonitor, the operator finds the rule, writes it
into a file for Prometheus and reloads it.
ruleSelectorNilUsesHelmValues: falseon Day 2 made that possible.
It takes one to two minutes until the operator has handed the rule to Prometheus. Then:
promtool query instant http://localhost:30090 'job:notes_http_requests:rate5m'Expected (the order changes):
job:notes_http_requests:rate5m{code="200", job="notes-api", method="GET"} => 4.30...
job:notes_http_requests:rate5m{code="201", job="notes-api", method="POST"} => 1.00...
The rule is now a perfectly normal time series with its own name.
There’s a reason GET is above 4: the rule doesn’t keep the
label path, so /healthz and
/metrics count too.
Save your progress? Sign up for free
Checkpoint
promtool query instant http://localhost:30090 \
'job:notes_http_request_duration_seconds:p95_5m < 0.1'Expected:
job:notes_http_request_duration_seconds:p95_5m{job="notes-api"} => 0.0096... @[...]
A comparison like < 0.1 filters: the time series only
remains if it meets the condition. If it shows up, the delay is gone.
After the rollout it takes five minutes until the window is clean
again.
kubectl -n app get prometheusruleExpected:
NAME AGE
notes-api-records 61s
Challenge
What share of all requests to notes-api is a POST? Use
the recording rule.
Hint: Two sum(...) with a
/ in between.
Solution
promtool query instant http://localhost:30090 \
'sum(job:notes_http_requests:rate5m{method="POST"})
/ sum(job:notes_http_requests:rate5m)'Expected (roughly):
{} => 0.18902842035690678 @[...]
One in five requests. It’s not exactly 0.2 because
/healthz and /metrics count in the
denominator. With {path="/notes"} in the rule this could be
changed – but the rule didn’t keep the label path.
Save your progress? Sign up for free
Enterprise Note
- Rate before sum:
sum(rate(x[5m]))is correct,rate(sum(x)[5m])is not. The second form takes every restart for a reset of the sum and reports spikes that never happened. - Recording rules for dashboards: A dashboard with 20 panels and ten users otherwise fires 200 queries per refresh. Precomputed time series are cheap.
- Native histograms: Prometheus 3 can store histograms with automatically fine buckets. They solve the problem from Drill 5 but require changes in the code and in the scrape. This book sticks with classic buckets.