Skip to content
Hands-on IT IT Drills

Day 3: PromQL and Recording Rules

Goal: You calculate rates from counters and percentiles from histograms, and store frequent queries as recording rules. · Duration: approx. 60 min · Prerequisite: Day 2

Briefing

  • PromQL is the query language of Prometheus. A name like notes_created_total returns an instant vector: the newest value per time series.
  • With [1m] after it, it becomes a range vector: all values of the last minute per time series. Functions like rate() turn that back into one number per time series.
  • Aggregations like sum by (method) combine time series and keep only the named labels.
  • A recording rule evaluates a query regularly and stores the result as a new time series. Dashboards and alerts then read the finished result.
 notes_http_requests_total   (counter, 4 time series per pod)
        │ rate(...[5m])      requests per second
        │ sum by (method)    across all pods and codes
        ▼
 {method="GET"} 4   {method="POST"} 1

Drill 1: Range Vector

promtool query instant http://localhost:30090 'notes_created_total[1m]'

Expected (shortened):

notes_created_total{..., pod="notes-api-57f4f47769-59ks2", ...} =>
247 @[1793866895.847]
257 @[1793866910.845]
263 @[1793866925.845]
271 @[1793866940.845]
...
  • Four values per pod, 15 seconds apart – the interval from the ServiceMonitor.
  • A range vector is raw material. Prometheus can’t draw it as a curve; you need a function that turns it into a number.

Save your progress? Sign up for free

Drill 2: rate()

promtool query instant http://localhost:30090 'rate(notes_created_total[5m])'

Expected (shortened):

{..., pod="notes-api-57f4f47769-59ks2", ...} => 0.45614035087719296 @[...]
{..., pod="notes-api-57f4f47769-m66nj", ...} => 0.43859649122807015 @[...]
{..., pod="notes-api-57f4f47769-gg2rr", ...} => 0.09 @[...]
  • rate() takes the values of the last five minutes and calculates how much the counter grew per second. Each pod creates just under half a note per second.
  • The third pod is the one from the Day 2 challenge. It’s already gone but still in the window of the last five minutes. After five minutes it disappears.
  • When a pod restarts, its counter drops to 0. rate() detects that and keeps calculating correctly. That’s why rate() belongs on every counter – never the bare number.
  • The window should cover at least four scrapes. At 15 seconds, [1m] is the lower limit, [5m] smooths better.

The metric name is missing in the output: after a calculation it is no longer a notes_created_total.

Save your progress? Sign up for free

Drill 3: Aggregating

promtool query instant http://localhost:30090 \
  'sum by (method) (rate(notes_http_requests_total{path="/notes"}[5m]))'

Expected:

{method="GET"} => 3.9719298245614034 @[...]
{method="POST"} => 0.983625730994152 @[...]

Four reads and one note per second – exactly what the load generator sends.

  • The order matters: first rate() per time series, then sum. The other way around, the restarts of individual pods would hide in the sum.
  • by (method) keeps only the label method. without (pod) would be the counterpart: keep all labels except pod.
  • {path="/notes"} is a selector. =~ allows regular expressions, such as code=~"5.." for all server errors.

Save your progress? Sign up for free

Drill 4: increase()

promtool query instant http://localhost:30090 \
  'sum(increase(notes_http_requests_total{method="POST"}[5m]))'

Expected (roughly):

{} => 302.1052631578947 @[...]

increase() is rate() times the window length: about 300 new notes in five minutes. That the result isn’t a round number is due to the extrapolation to the edge of the window.

Save your progress? Sign up for free

Drill 5: Percentiles

An average hides outliers. How long do the slowest five percent wait?

promtool query instant http://localhost:30090 \
  'histogram_quantile(0.95, sum by (le) (rate(
     notes_http_request_duration_seconds_bucket{path="/notes"}[5m])))'

Expected (roughly):

{} => 0.009103579487633288 @[...]
  • histogram_quantile(0.95, …) looks for the bucket in which the 95 percent mark lies and estimates linearly within it. 95 percent of requests are faster than about 9 ms.
  • sum by (le) is mandatory: the label le with the bucket limits must be preserved, everything else is combined.

Slow down notes-api as on Day 1:

kubectl -n app set env deploy/notes-api NOTES_DELAY=300ms
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

Wait five minutes – that’s how long the window is –, then the same query again:

promtool query instant http://localhost:30090 \
  'histogram_quantile(0.95, sum by (le) (rate(
     notes_http_request_duration_seconds_bucket{path="/notes"}[5m])))'

Expected (roughly):

{} => 0.4875 @[...]

Every request takes a little over 300 ms, the result says almost 490 ms. All values are in the bucket from 0.25 to 0.5 s, and within it histogram_quantile pretends they are evenly distributed. Percentiles from histograms are estimates. How good they are depends on the bucket limits.

Remove the delay:

kubectl -n app set env deploy/notes-api NOTES_DELAY-
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

Save your progress? Sign up for free

Drill 6: Recording Rules

The query from Drill 5 is long, and every dashboard would recalculate it on every load. A recording rule calculates it once every 30 seconds:

cd ~/obs-lab
cat > notes-rules.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: notes-api-records
  namespace: app
spec:
  groups:
    - name: notes-api.records
      interval: 30s
      rules:
        - record: job:notes_http_requests:rate5m
          expr: sum by (job, method, code) (rate(notes_http_requests_total[5m]))
        - record: job:notes_http_request_duration_seconds:p95_5m
          expr: |
            histogram_quantile(0.95, sum by (job, le) (
              rate(notes_http_request_duration_seconds_bucket{path="/notes"}[5m])))
EOF
kubectl apply -f notes-rules.yaml

Expected:

prometheusrule.monitoring.coreos.com/notes-api-records created
  • The name follows the convention level:metric:operation. job: says that it is aggregated across all pods of a job. :rate5m describes the calculation.
  • by (job, …) keeps job on purpose. That way it stays clear where the number comes from when more applications get the same rule later.
  • As with the ServiceMonitor, the operator finds the rule, writes it into a file for Prometheus and reloads it. ruleSelectorNilUsesHelmValues: false on Day 2 made that possible.

It takes one to two minutes until the operator has handed the rule to Prometheus. Then:

promtool query instant http://localhost:30090 'job:notes_http_requests:rate5m'

Expected (the order changes):

job:notes_http_requests:rate5m{code="200", job="notes-api", method="GET"} => 4.30...
job:notes_http_requests:rate5m{code="201", job="notes-api", method="POST"} => 1.00...

The rule is now a perfectly normal time series with its own name. There’s a reason GET is above 4: the rule doesn’t keep the label path, so /healthz and /metrics count too.

Save your progress? Sign up for free

Checkpoint

promtool query instant http://localhost:30090 \
  'job:notes_http_request_duration_seconds:p95_5m < 0.1'

Expected:

job:notes_http_request_duration_seconds:p95_5m{job="notes-api"} => 0.0096... @[...]

A comparison like < 0.1 filters: the time series only remains if it meets the condition. If it shows up, the delay is gone. After the rollout it takes five minutes until the window is clean again.

kubectl -n app get prometheusrule

Expected:

NAME                AGE
notes-api-records   61s

Challenge

What share of all requests to notes-api is a POST? Use the recording rule.

Hint: Two sum(...) with a / in between.

Solution

promtool query instant http://localhost:30090 \
  'sum(job:notes_http_requests:rate5m{method="POST"})
     / sum(job:notes_http_requests:rate5m)'

Expected (roughly):

{} => 0.18902842035690678 @[...]

One in five requests. It’s not exactly 0.2 because /healthz and /metrics count in the denominator. With {path="/notes"} in the rule this could be changed – but the rule didn’t keep the label path.

Save your progress? Sign up for free

Enterprise Note

  • Rate before sum: sum(rate(x[5m])) is correct, rate(sum(x)[5m]) is not. The second form takes every restart for a reset of the sum and reports spikes that never happened.
  • Recording rules for dashboards: A dashboard with 20 panels and ten users otherwise fires 200 queries per refresh. Precomputed time series are cheap.
  • Native histograms: Prometheus 3 can store histograms with automatically fine buckets. They solve the problem from Drill 5 but require changes in the code and in the scrape. This book sticks with classic buckets.