Day 1: Alerts Are Born
Goal: Prometheus collects metrics from notes-api, evaluates three alerting rules, and you watch an alert go from inactive through pending to firing. · Duration: approx. 50 min · Prerequisite: Day 0
Briefing
- Alertmanager does not create alerts. Prometheus does that: an alerting rule is a PromQL query. If it returns time series, the alert is active, one time series per alert.
for: 1mmeans: the condition must hold continuously for one minute. Until then, the alert is pending, afterwards firing. Only firing alerts are sent to Alertmanager.- Labels on the rule (
severity,team) later decide where an alert goes. Annotations (summary,runbook_url) are text for humans. - If you do not know Prometheus yet: “Observability hands-on” explains scrapes and PromQL in detail. Here, the essentials are enough.
notes-api /metrics ◄── Scrape every 15s ── Prometheus :9090
node-exporter ◄──────────────────────┘ │
rules every 15 s
│
inactive ─► pending ─(for)─► firing
Drill 1: promtool
promtool checks configuration and rules before
Prometheus loads them, and runs queries from the command line. It is
included in the Prometheus archive:
cd ~/am-lab
curl -fsSLO https://github.com/prometheus/prometheus/releases/download/\
v3.16.0/prometheus-3.16.0.linux-amd64.tar.gz
tar xzf prometheus-3.16.0.linux-amd64.tar.gz
mv prometheus-3.16.0.linux-amd64/promtool ~/.local/bin/
rm -r prometheus-3.16.0.linux-amd64*
promtool --version | head -1Expected:
promtool, version 3.16.0 (branch: HEAD, revision: e2e28adeee2c5883d000fc82aca6008cfbf7e96d)
Prometheus itself will run as a container shortly. From the archive, you only need the tool.
Save your progress? Sign up for free
Drill 2: Prometheus Configuration
mkdir -p prometheus/rules
cat > prometheus/prometheus.yml <<'EOF'
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
env: prod
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["localhost:9090"]
- job_name: notes-api
static_configs:
- targets: ["notes-api:8080"]
- job_name: node
static_configs:
- targets: ["node-exporter:9100"]
EOFevaluation_interval: 15s: this is how often Prometheus evaluates all rules. The default is one minute; in the lab you want to see results faster.external_labels: Prometheus attaches this label to every alert it sends. On Day 9, a second environmentdevjoins. Theenvlabel then tells you where an alert comes from.node-exporterprovides metrics of the VM itself. You need them from Day 3 on.
Save your progress? Sign up for free
Drill 3: The Rules
cat > prometheus/rules/notes.yml <<'EOF'
groups:
- name: notes
rules:
- alert: NotesApiDown
expr: up{job="notes-api"} == 0
for: 1m
labels:
severity: critical
team: notes
annotations:
summary: "notes-api auf {{ $labels.instance }} antwortet nicht"
runbook_url: "https://wiki.example.com/runbooks/notes-api-down"
- alert: NotesDbDown
expr: notes_db_up == 0
for: 1m
labels:
severity: critical
team: notes
annotations:
summary: "notes-api erreicht Postgres nicht"
- alert: NotesHighErrorRate
expr: |
sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[2m]))
/ sum(rate(notes_http_requests_total{path="/notes"}[2m])) > 0.05
for: 2m
labels:
severity: warning
team: notes
annotations:
summary: "{{ $value | humanizePercentage }} der Anfragen an /notes scheitern"
EOF
promtool check rules prometheus/rules/notes.ymlExpected:
Checking prometheus/rules/notes.yml
SUCCESS: 3 rules found
upis a metric that Prometheus itself writes for every target: 1 if the scrape succeeded, otherwise 0.- Templates are allowed in annotations.
$labelsare the labels of the time series,$valueits value.humanizePercentageturns0.18into the text18%. - The error rate has no
instancelabel becausesum()merges all time series. What remains after an aggregation determines the labels of the alert.
Save your progress? Sign up for free
Drill 4: Start Prometheus
The second building block, monitoring.yaml:
cat > monitoring.yaml <<'EOF'
services:
prometheus:
image: prom/prometheus:v3.16.0
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --web.enable-lifecycle
- --web.external-url=http://${VM_IP}:9090
volumes:
- ./prometheus:/etc/prometheus:ro
- promdata:/prometheus
ports:
- "9090:9090"
node-exporter:
image: prom/node-exporter:v1.12.1
volumes:
promdata:
EOF
echo " - monitoring.yaml" >> compose.yaml
docker compose up -d--web.enable-lifecycleallows reloading the configuration over HTTP:curl -X POST localhost:9090/-/reload. Without the flag, you would have to restart the container.--web.external-urlis the address under which humans reach Prometheus. Prometheus builds the links in every alert from it. Without the flag, the container name would appear there, and no browser would find it.${VM_IP}comes from.env.
After about 20 seconds:
promtool query instant http://localhost:9090 upExpected:
up{instance="localhost:9090", job="prometheus"} => 1 @[1794856598.129]
up{instance="notes-api:8080", job="notes-api"} => 1 @[1794856598.129]
up{instance="node-exporter:9100", job="node"} => 1 @[1794856598.129]
Three targets, all at 1. The web UI is at
http://172.16.0.61:9090, and the Alerts
page shows your rules.
Save your progress? Sign up for free
Drill 5: An Alert Is Born
Stop notes-api:
docker compose stop notes-apiAfter 30 seconds at most, the alert is pending:
promtool query instant http://localhost:9090 ALERTSExpected:
ALERTS{alertname="NotesApiDown", alertstate="pending", instance="notes-api:8080", job="notes-api", severity="critical", team="notes"} => 1 @[1794855560.037]
ALERTS is a time series that Prometheus writes for every
active alert. The alertstate label tells you where it
stands. The labels of the rule are already attached. Wait another minute
and query again:
promtool query instant http://localhost:9090 ALERTSExpected:
ALERTS{alertname="NotesApiDown", alertstate="firing", instance="notes-api:8080", job="notes-api", severity="critical", team="notes"} => 1 @[1794855617.674]
The same information as JSON, with the time since which the condition has held:
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[0]'Expected (shortened):
{
"labels": {
"alertname": "NotesApiDown",
"instance": "notes-api:8080",
"job": "notes-api",
"severity": "critical",
"team": "notes"
},
"state": "firing",
"activeAt": "2026-11-16T18:59:13.78407747Z",
…
}The env label is still missing here. Prometheus only
attaches external_labels when sending. Start notes-api
again:
docker compose start notes-apiAfter the next scrape, the alert disappears.
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts | length'
prints 0.
Tried it: If
pendingis still there after more than a minute, do the math.forcounts fromactiveAt, and Prometheus only checks every 15 seconds. In the test, the alert became firing 64 seconds afteractiveAt.
Save your progress? Sign up for free
Drill 6: Fault Instead of Outage
The error rate does not need an outage, only bad responses:
NOTES_ERROR_RATE=0.3 docker compose up -d notes-apiAfter a little over two minutes:
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[0].annotations'Expected:
{
"summary": "31.28% der Anfragen an /notes scheitern"
}The value comes from $value and changes with every
evaluation. Back to the normal state:
docker compose up -d notes-apiSave your progress? Sign up for free
Checkpoint
promtool check rules prometheus/rules/notes.ymlExpected: SUCCESS: 3 rules found.
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'Expected: NotesApiDown, NotesDbDown and
NotesHighErrorRate, one line each.
promtool query instant http://localhost:9090 notes_db_upExpected: one time series with => 1.
Challenge
Write a fourth rule NotesSlow (severity
warning, team notes): it fires when 95 % of
the requests to /notes take longer than 250 milliseconds,
for one minute. Trigger it with NOTES_DELAY.
Note: notes-api provides the histogram
notes_http_request_duration_seconds_bucket.
histogram_quantile(0.95, …) needs the rates per bucket,
aggregated by (le). Reload with
curl -X POST localhost:9090/-/reload.
Solution
cat >> prometheus/rules/notes.yml <<'EOF'
- alert: NotesSlow
expr: |
histogram_quantile(0.95,
sum by (le) (rate(notes_http_request_duration_seconds_bucket{path="/notes"}[2m]))
) > 0.25
for: 1m
labels:
severity: warning
team: notes
annotations:
summary: "95 % der Anfragen an /notes dauern bis zu {{ $value | humanizeDuration }}"
EOF
promtool check rules prometheus/rules/notes.yml
curl -s -X POST localhost:9090/-/reload
NOTES_DELAY=300ms docker compose up -d notes-apiExpected: SUCCESS: 4 rules found. After about two
minutes:
curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[].annotations.summary'Expected:
95 % der Anfragen an /notes dauern bis zu 487.5ms
The requests take 300 milliseconds, the alert reports 487.5. Prometheus only knows the bucket boundaries, here 0.25 and 0.5 seconds, and interpolates linearly in between. That is good enough for the threshold, but rough for the display. Back to the normal state:
docker compose up -d notes-apiThe NotesSlow rule stays in the file for the next
days.
Save your progress? Sign up for free
Enterprise Note
- Every alert needs a
runbook_url. Whoever is woken up at night should not have to search for what to do first. - Rules belong in Git and through
promtool check rulesin CI. For rules with logic, there are unit tests:promtool test rules. - Alert on symptoms that users feel (error rate, latency), not on
every cause. Cause alerts like
NotesDbDownhelp with diagnosis and get a quieter voice on Day 5. foris a compromise: too short, and every outlier wakes someone up. Too long, and users notice the outage before you do.