Day 12: SLOs and Burn Rate Alerts
Goal: A service level objective for notes-api, an error budget you can calculate, and alerts that trigger based on budget consumption instead of fixed thresholds. · Duration: approx. 75 min · Prerequisite: Day 11
Briefing
- An SLI (service level indicator) measures what
users feel – for example the share of successful requests. An
SLO (objective) is the target for it: “99.5 percent of
requests to
/notessucceed, measured over 30 days.” - The missing 0.5 percent is the error budget. Out of 1000 requests, five may fail without breaking the target. The budget is there to be spent – on deployments, experiments, bad luck.
- The burn rate says how fast the budget melts away. Burn rate 1 means: at the end of the 30 days, it’s used up exactly. Burn rate 14.4 means: it’s gone in two days.
- Good SLO alerts check two windows at the same time: a long one, so that a short hiccup doesn’t wake anyone, and a short one, so that the alert ends as soon as the problem is over.
error rate ──► ÷ 0.005 (budget) ──► burn rate
> 14.4 in 1 h and 5 min ──► critical (pager)
> 6 in 6 h and 30 min ──► warning (ticket)
Drill 1: The SLI
promtool query instant http://localhost:30090 '
sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[5m]))
/ sum(rate(notes_http_requests_total{path="/notes"}[5m]))'Expected: no output. If there’s a small number, the errors from the Day 11 challenge are still lingering – wait five minutes and ask again.
Without errors there is no time series with code=~"5..".
The sum over nothing is nothing, and nothing divided by something stays
nothing. For an SLI that’s wrong – it should be 0.
or vector(0) inserts a 0 when the left side is empty:
promtool query instant http://localhost:30090 '
(sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[5m])) or vector(0))
/ sum(rate(notes_http_requests_total{path="/notes"}[5m]))'Expected:
{} => 0 @[...]
path="/notes": the SLO applies to what users call./healthzand/metricswould make the ratio look better than it is.- Only server errors count. A 401 without an API key isn’t a failure of notes-api but a mistake by the caller.
Save your progress? Sign up for free
Drill 2: Rules for All Windows
The burn rate alerts need the error rate over four time ranges. Each gets a recording rule, and the alerts build on them:
cd ~/obs-lab
cat > notes-slo.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: notes-api-slo
namespace: app
spec:
groups:
- name: notes-api.slo
interval: 30s
rules:
- record: notes:sli_errors:ratio_rate5m
expr: |
(sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[5m]))
or vector(0))
/ sum(rate(notes_http_requests_total{path="/notes"}[5m]))
- record: notes:sli_errors:ratio_rate30m
expr: |
(sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[30m]))
or vector(0))
/ sum(rate(notes_http_requests_total{path="/notes"}[30m]))
- record: notes:sli_errors:ratio_rate1h
expr: |
(sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[1h]))
or vector(0))
/ sum(rate(notes_http_requests_total{path="/notes"}[1h]))
- record: notes:sli_errors:ratio_rate6h
expr: |
(sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[6h]))
or vector(0))
/ sum(rate(notes_http_requests_total{path="/notes"}[6h]))
- alert: NotesSloFastBurn
expr: |
notes:sli_errors:ratio_rate1h > (14.4 * 0.005)
and notes:sli_errors:ratio_rate5m > (14.4 * 0.005)
for: 2m
labels:
severity: critical
namespace: app
annotations:
summary: "notes-api verbrennt das Fehlerbudget 14-mal zu schnell"
- alert: NotesSloSlowBurn
expr: |
notes:sli_errors:ratio_rate6h > (6 * 0.005)
and notes:sli_errors:ratio_rate30m > (6 * 0.005)
for: 15m
labels:
severity: warning
namespace: app
annotations:
summary: "notes-api verbrennt das Fehlerbudget 6-mal zu schnell"
EOF
kubectl apply -f notes-slo.yamlExpected:
prometheusrule.monitoring.coreos.com/notes-api-slo created
0.005is the error budget of an SLO of 99.5 percent.14.4 * 0.005= 7.2 percent error rate – at this pace, 2 percent of the monthly budget would be gone in one hour.- FastBurn catches outages: error rate above 7.2 percent in the last hour and in the last five minutes. The short window lets the alert go out quickly once the problem is fixed.
- SlowBurn catches creeping degradation: above 3 percent in six hours and in 30 minutes. That goes into a ticket, not to the pager.
- The factors 14.4 and 6 come from Google’s SRE Workbook, chapter “Alerting on SLOs”. They are a proven starting point, not a law of nature.
After two minutes:
promtool query instant http://localhost:30090 '{__name__=~"notes:sli_errors:.*"}'Expected (the order changes):
notes:sli_errors:ratio_rate5m => 0 @[...]
notes:sli_errors:ratio_rate30m => 0.013521214149718465 @[...]
notes:sli_errors:ratio_rate1h => 0.013521214149718465 @[...]
notes:sli_errors:ratio_rate6h => 0.013521214149718465 @[...]
The long windows aren’t 0: they remember the faults from Day 11. That 30 minutes, one hour and six hours show the same value is due to Day 11: the upgrade there restarted Prometheus, and without a PersistentVolume its memory only begins at that point. All three windows see the same data.
Save your progress? Sign up for free
Drill 3: Burn Rate and Remaining Budget
The current burn rate over one hour:
promtool query instant http://localhost:30090 'notes:sli_errors:ratio_rate1h / 0.005'Expected (roughly):
{} => 2.704242829943693 @[...]
Above 1: at this pace the budget would be gone after eleven days (30 ÷ 2.7). Without the practice faults the value would be 0.
How much of the budget is left? A real SLO calculates over 30 days. In the lab, Prometheus keeps only three, so here is the budget for one day:
promtool query instant http://localhost:30090 '
1 - ((sum(increase(notes_http_requests_total{path="/notes",code=~"5.."}[1d]))
or vector(0))
/ sum(increase(notes_http_requests_total{path="/notes"}[1d]))) / 0.005'Expected (roughly):
{} => -1.5555480108446664 @[...]
Negative: the daily budget is overdrawn. No wonder – since the restart of Prometheus on Day 11, you have produced errors on purpose. In production, this would be the moment to put new features on hold and invest in stability. That’s exactly what the budget is for: a number teams can make decisions on.
Save your progress? Sign up for free
Drill 4: An Outage
All requests to /notes fail:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=1
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
Watch the burn rate over both windows of the FastBurn alert. After two minutes:
promtool query instant http://localhost:30090 'notes:sli_errors:ratio_rate5m / 0.005'
promtool query instant http://localhost:30090 'notes:sli_errors:ratio_rate1h / 0.005'Expected (the numbers grow with every minute):
{} => 76.11279107372314 @[...]
{} => 36.850068908800594 @[...]
The short window heads toward 200 (100 percent errors ÷ 0.5 percent
budget), the long one slowly catches up. As soon as both are above 14.4,
for: 2m starts. How quickly the long window crosses the
threshold depends on how much error-free data it has from the last hour
– with a full hour without errors, after a good four minutes.
amtool alert query alertname=NotesSloFastBurnExpected (after three to eight minutes):
Alertname Starts At Summary...
NotesSloFastBurn 2026-11-05 09:55:50 UTC notes-api verbrennt das Fehlerbudget 14-ma...
A total outage wakes someone up after a few minutes. An error rate of 2 percent never triggers FastBurn – it only burns the budget four times too fast. That’s what SlowBurn is for.
Save your progress? Sign up for free
Drill 5: All-Clear
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
The long window stays above 14.4 for a while, but the short one drops below the threshold after five minutes. Because both conditions must hold, the alert goes out:
amtool alert query alertname=NotesSloFastBurnExpected (after about five minutes): only the header line.
That’s the reason for two windows. With only the long one, the alert would keep firing for an hour after the fix.
Save your progress? Sign up for free
Checkpoint
kubectl -n app get prometheusruleExpected:
NAME AGE
notes-api-alerts 102m
notes-api-records 104m
notes-api-slo 26m
notes-api-slow 85m
promtool query instant http://localhost:30090 'count({__name__=~"notes:sli_errors:.*"})'Expected:
{} => 4 @[...]
Four windows, one time series each.
Challenge
Formulate a second SLI for the response time: the
share of requests to GET /notes that are answered within
250 ms. How high is it right now?
Hint: The histogram has a bucket
le="0.25". _count is the total.
Solution
promtool query instant http://localhost:30090 '
sum(rate(notes_http_request_duration_seconds_bucket{
method="GET",path="/notes",le="0.25"}[1h]))
/ sum(rate(notes_http_request_duration_seconds_count{
method="GET",path="/notes"}[1h]))'Expected (roughly):
{} => 0.9736396162752308 @[...]
97 percent of the requests in the last hour were fast enough. The
remaining three percent come from Drill 3 on Day 11 with
NOTES_DELAY=400ms. Because the bucket sits exactly at 250
ms, this SLI is exact – no estimating as with
histogram_quantile. That’s why latency SLO limits are
placed on a bucket boundary.
Save your progress? Sign up for free
Enterprise Note
- SLOs as code: Sloth and Pyrra generate all recording rules and burn rate alerts of this chapter from a few lines of YAML. Pyrra comes with a UI that shows remaining budgets.
- The target comes from the user: 99.5, 99.9 or 99.99 percent is a business decision, not a technical one. Every additional nine costs many times more. An SLO that is never breached is too low – or it measures the wrong thing.
- Budget policy: An SLO only has an effect with an agreement on what happens when the budget is used up: deploy freeze, work on stability, retrospective. Without it, it’s just another curve.