Skip to content
Hands-on IT IT Drills

Day 12: SLOs and Burn Rate Alerts

Goal: A service level objective for notes-api, an error budget you can calculate, and alerts that trigger based on budget consumption instead of fixed thresholds. · Duration: approx. 75 min · Prerequisite: Day 11

Briefing

  • An SLI (service level indicator) measures what users feel – for example the share of successful requests. An SLO (objective) is the target for it: “99.5 percent of requests to /notes succeed, measured over 30 days.”
  • The missing 0.5 percent is the error budget. Out of 1000 requests, five may fail without breaking the target. The budget is there to be spent – on deployments, experiments, bad luck.
  • The burn rate says how fast the budget melts away. Burn rate 1 means: at the end of the 30 days, it’s used up exactly. Burn rate 14.4 means: it’s gone in two days.
  • Good SLO alerts check two windows at the same time: a long one, so that a short hiccup doesn’t wake anyone, and a short one, so that the alert ends as soon as the problem is over.
 error rate ──► ÷ 0.005 (budget) ──► burn rate
                                       > 14.4 in 1 h and 5 min  ──► critical (pager)
                                       > 6    in 6 h and 30 min ──► warning  (ticket)

Drill 1: The SLI

promtool query instant http://localhost:30090 '
  sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[5m]))
    / sum(rate(notes_http_requests_total{path="/notes"}[5m]))'

Expected: no output. If there’s a small number, the errors from the Day 11 challenge are still lingering – wait five minutes and ask again.

Without errors there is no time series with code=~"5..". The sum over nothing is nothing, and nothing divided by something stays nothing. For an SLI that’s wrong – it should be 0. or vector(0) inserts a 0 when the left side is empty:

promtool query instant http://localhost:30090 '
  (sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[5m])) or vector(0))
    / sum(rate(notes_http_requests_total{path="/notes"}[5m]))'

Expected:

{} => 0 @[...]
  • path="/notes": the SLO applies to what users call. /healthz and /metrics would make the ratio look better than it is.
  • Only server errors count. A 401 without an API key isn’t a failure of notes-api but a mistake by the caller.

Save your progress? Sign up for free

Drill 2: Rules for All Windows

The burn rate alerts need the error rate over four time ranges. Each gets a recording rule, and the alerts build on them:

cd ~/obs-lab
cat > notes-slo.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: notes-api-slo
  namespace: app
spec:
  groups:
    - name: notes-api.slo
      interval: 30s
      rules:
        - record: notes:sli_errors:ratio_rate5m
          expr: |
            (sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[5m]))
               or vector(0))
              / sum(rate(notes_http_requests_total{path="/notes"}[5m]))
        - record: notes:sli_errors:ratio_rate30m
          expr: |
            (sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[30m]))
               or vector(0))
              / sum(rate(notes_http_requests_total{path="/notes"}[30m]))
        - record: notes:sli_errors:ratio_rate1h
          expr: |
            (sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[1h]))
               or vector(0))
              / sum(rate(notes_http_requests_total{path="/notes"}[1h]))
        - record: notes:sli_errors:ratio_rate6h
          expr: |
            (sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[6h]))
               or vector(0))
              / sum(rate(notes_http_requests_total{path="/notes"}[6h]))
        - alert: NotesSloFastBurn
          expr: |
            notes:sli_errors:ratio_rate1h > (14.4 * 0.005)
              and notes:sli_errors:ratio_rate5m > (14.4 * 0.005)
          for: 2m
          labels:
            severity: critical
            namespace: app
          annotations:
            summary: "notes-api verbrennt das Fehlerbudget 14-mal zu schnell"
        - alert: NotesSloSlowBurn
          expr: |
            notes:sli_errors:ratio_rate6h > (6 * 0.005)
              and notes:sli_errors:ratio_rate30m > (6 * 0.005)
          for: 15m
          labels:
            severity: warning
            namespace: app
          annotations:
            summary: "notes-api verbrennt das Fehlerbudget 6-mal zu schnell"
EOF
kubectl apply -f notes-slo.yaml

Expected:

prometheusrule.monitoring.coreos.com/notes-api-slo created
  • 0.005 is the error budget of an SLO of 99.5 percent. 14.4 * 0.005 = 7.2 percent error rate – at this pace, 2 percent of the monthly budget would be gone in one hour.
  • FastBurn catches outages: error rate above 7.2 percent in the last hour and in the last five minutes. The short window lets the alert go out quickly once the problem is fixed.
  • SlowBurn catches creeping degradation: above 3 percent in six hours and in 30 minutes. That goes into a ticket, not to the pager.
  • The factors 14.4 and 6 come from Google’s SRE Workbook, chapter “Alerting on SLOs”. They are a proven starting point, not a law of nature.

After two minutes:

promtool query instant http://localhost:30090 '{__name__=~"notes:sli_errors:.*"}'

Expected (the order changes):

notes:sli_errors:ratio_rate5m => 0 @[...]
notes:sli_errors:ratio_rate30m => 0.013521214149718465 @[...]
notes:sli_errors:ratio_rate1h => 0.013521214149718465 @[...]
notes:sli_errors:ratio_rate6h => 0.013521214149718465 @[...]

The long windows aren’t 0: they remember the faults from Day 11. That 30 minutes, one hour and six hours show the same value is due to Day 11: the upgrade there restarted Prometheus, and without a PersistentVolume its memory only begins at that point. All three windows see the same data.

Save your progress? Sign up for free

Drill 3: Burn Rate and Remaining Budget

The current burn rate over one hour:

promtool query instant http://localhost:30090 'notes:sli_errors:ratio_rate1h / 0.005'

Expected (roughly):

{} => 2.704242829943693 @[...]

Above 1: at this pace the budget would be gone after eleven days (30 ÷ 2.7). Without the practice faults the value would be 0.

How much of the budget is left? A real SLO calculates over 30 days. In the lab, Prometheus keeps only three, so here is the budget for one day:

promtool query instant http://localhost:30090 '
  1 - ((sum(increase(notes_http_requests_total{path="/notes",code=~"5.."}[1d]))
          or vector(0))
         / sum(increase(notes_http_requests_total{path="/notes"}[1d]))) / 0.005'

Expected (roughly):

{} => -1.5555480108446664 @[...]

Negative: the daily budget is overdrawn. No wonder – since the restart of Prometheus on Day 11, you have produced errors on purpose. In production, this would be the moment to put new features on hold and invest in stability. That’s exactly what the budget is for: a number teams can make decisions on.

Save your progress? Sign up for free

Drill 4: An Outage

All requests to /notes fail:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=1
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

Watch the burn rate over both windows of the FastBurn alert. After two minutes:

promtool query instant http://localhost:30090 'notes:sli_errors:ratio_rate5m / 0.005'
promtool query instant http://localhost:30090 'notes:sli_errors:ratio_rate1h / 0.005'

Expected (the numbers grow with every minute):

{} => 76.11279107372314 @[...]
{} => 36.850068908800594 @[...]

The short window heads toward 200 (100 percent errors ÷ 0.5 percent budget), the long one slowly catches up. As soon as both are above 14.4, for: 2m starts. How quickly the long window crosses the threshold depends on how much error-free data it has from the last hour – with a full hour without errors, after a good four minutes.

amtool alert query alertname=NotesSloFastBurn

Expected (after three to eight minutes):

Alertname         Starts At                Summary...
NotesSloFastBurn  2026-11-05 09:55:50 UTC  notes-api verbrennt das Fehlerbudget 14-ma...

A total outage wakes someone up after a few minutes. An error rate of 2 percent never triggers FastBurn – it only burns the budget four times too fast. That’s what SlowBurn is for.

Save your progress? Sign up for free

Drill 5: All-Clear

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

The long window stays above 14.4 for a while, but the short one drops below the threshold after five minutes. Because both conditions must hold, the alert goes out:

amtool alert query alertname=NotesSloFastBurn

Expected (after about five minutes): only the header line.

That’s the reason for two windows. With only the long one, the alert would keep firing for an hour after the fix.

Save your progress? Sign up for free

Checkpoint

kubectl -n app get prometheusrule

Expected:

NAME                AGE
notes-api-alerts    102m
notes-api-records   104m
notes-api-slo       26m
notes-api-slow      85m
promtool query instant http://localhost:30090 'count({__name__=~"notes:sli_errors:.*"})'

Expected:

{} => 4 @[...]

Four windows, one time series each.

Challenge

Formulate a second SLI for the response time: the share of requests to GET /notes that are answered within 250 ms. How high is it right now?

Hint: The histogram has a bucket le="0.25". _count is the total.

Solution

promtool query instant http://localhost:30090 '
  sum(rate(notes_http_request_duration_seconds_bucket{
        method="GET",path="/notes",le="0.25"}[1h]))
    / sum(rate(notes_http_request_duration_seconds_count{
        method="GET",path="/notes"}[1h]))'

Expected (roughly):

{} => 0.9736396162752308 @[...]

97 percent of the requests in the last hour were fast enough. The remaining three percent come from Drill 3 on Day 11 with NOTES_DELAY=400ms. Because the bucket sits exactly at 250 ms, this SLI is exact – no estimating as with histogram_quantile. That’s why latency SLO limits are placed on a bucket boundary.

Save your progress? Sign up for free

Enterprise Note

  • SLOs as code: Sloth and Pyrra generate all recording rules and burn rate alerts of this chapter from a few lines of YAML. Pyrra comes with a UI that shows remaining budgets.
  • The target comes from the user: 99.5, 99.9 or 99.99 percent is a business decision, not a technical one. Every additional nine costs many times more. An SLO that is never breached is too low – or it measures the wrong thing.
  • Budget policy: An SLO only has an effect with an agreement on what happens when the budget is used up: deploy freeze, work on stability, retrospective. Without it, it’s just another curve.