Skip to content
Hands-on IT IT Drills

Day 1: Alerts Are Born

Goal: Prometheus collects metrics from notes-api, evaluates three alerting rules, and you watch an alert go from inactive through pending to firing. · Duration: approx. 50 min · Prerequisite: Day 0

Briefing

  • Alertmanager does not create alerts. Prometheus does that: an alerting rule is a PromQL query. If it returns time series, the alert is active, one time series per alert.
  • for: 1m means: the condition must hold continuously for one minute. Until then, the alert is pending, afterwards firing. Only firing alerts are sent to Alertmanager.
  • Labels on the rule (severity, team) later decide where an alert goes. Annotations (summary, runbook_url) are text for humans.
  • If you do not know Prometheus yet: “Observability hands-on” explains scrapes and PromQL in detail. Here, the essentials are enough.
 notes-api /metrics ◄── Scrape every 15s ── Prometheus :9090
 node-exporter      ◄──────────────────────┘    │
                                          rules every 15 s
                                                │
                              inactive ─► pending ─(for)─► firing

Drill 1: promtool

promtool checks configuration and rules before Prometheus loads them, and runs queries from the command line. It is included in the Prometheus archive:

cd ~/am-lab
curl -fsSLO https://github.com/prometheus/prometheus/releases/download/\
v3.16.0/prometheus-3.16.0.linux-amd64.tar.gz
tar xzf prometheus-3.16.0.linux-amd64.tar.gz
mv prometheus-3.16.0.linux-amd64/promtool ~/.local/bin/
rm -r prometheus-3.16.0.linux-amd64*
promtool --version | head -1

Expected:

promtool, version 3.16.0 (branch: HEAD, revision: e2e28adeee2c5883d000fc82aca6008cfbf7e96d)

Prometheus itself will run as a container shortly. From the archive, you only need the tool.

Save your progress? Sign up for free

Drill 2: Prometheus Configuration

mkdir -p prometheus/rules
cat > prometheus/prometheus.yml <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    env: prod

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: notes-api
    static_configs:
      - targets: ["notes-api:8080"]
  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]
EOF
  • evaluation_interval: 15s: this is how often Prometheus evaluates all rules. The default is one minute; in the lab you want to see results faster.
  • external_labels: Prometheus attaches this label to every alert it sends. On Day 9, a second environment dev joins. The env label then tells you where an alert comes from.
  • node-exporter provides metrics of the VM itself. You need them from Day 3 on.

Save your progress? Sign up for free

Drill 3: The Rules

cat > prometheus/rules/notes.yml <<'EOF'
groups:
  - name: notes
    rules:
      - alert: NotesApiDown
        expr: up{job="notes-api"} == 0
        for: 1m
        labels:
          severity: critical
          team: notes
        annotations:
          summary: "notes-api auf {{ $labels.instance }} antwortet nicht"
          runbook_url: "https://wiki.example.com/runbooks/notes-api-down"
      - alert: NotesDbDown
        expr: notes_db_up == 0
        for: 1m
        labels:
          severity: critical
          team: notes
        annotations:
          summary: "notes-api erreicht Postgres nicht"
      - alert: NotesHighErrorRate
        expr: |
          sum(rate(notes_http_requests_total{path="/notes",code=~"5.."}[2m]))
            / sum(rate(notes_http_requests_total{path="/notes"}[2m])) > 0.05
        for: 2m
        labels:
          severity: warning
          team: notes
        annotations:
          summary: "{{ $value | humanizePercentage }} der Anfragen an /notes scheitern"
EOF
promtool check rules prometheus/rules/notes.yml

Expected:

Checking prometheus/rules/notes.yml
  SUCCESS: 3 rules found
  • up is a metric that Prometheus itself writes for every target: 1 if the scrape succeeded, otherwise 0.
  • Templates are allowed in annotations. $labels are the labels of the time series, $value its value. humanizePercentage turns 0.18 into the text 18%.
  • The error rate has no instance label because sum() merges all time series. What remains after an aggregation determines the labels of the alert.

Save your progress? Sign up for free

Drill 4: Start Prometheus

The second building block, monitoring.yaml:

cat > monitoring.yaml <<'EOF'
services:
  prometheus:
    image: prom/prometheus:v3.16.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.enable-lifecycle
      - --web.external-url=http://${VM_IP}:9090
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - promdata:/prometheus
    ports:
      - "9090:9090"

  node-exporter:
    image: prom/node-exporter:v1.12.1

volumes:
  promdata:
EOF
echo "  - monitoring.yaml" >> compose.yaml
docker compose up -d
  • --web.enable-lifecycle allows reloading the configuration over HTTP: curl -X POST localhost:9090/-/reload. Without the flag, you would have to restart the container.
  • --web.external-url is the address under which humans reach Prometheus. Prometheus builds the links in every alert from it. Without the flag, the container name would appear there, and no browser would find it. ${VM_IP} comes from .env.

After about 20 seconds:

promtool query instant http://localhost:9090 up

Expected:

up{instance="localhost:9090", job="prometheus"} => 1 @[1794856598.129]
up{instance="notes-api:8080", job="notes-api"} => 1 @[1794856598.129]
up{instance="node-exporter:9100", job="node"} => 1 @[1794856598.129]

Three targets, all at 1. The web UI is at http://172.16.0.61:9090, and the Alerts page shows your rules.

Save your progress? Sign up for free

Drill 5: An Alert Is Born

Stop notes-api:

docker compose stop notes-api

After 30 seconds at most, the alert is pending:

promtool query instant http://localhost:9090 ALERTS

Expected:

ALERTS{alertname="NotesApiDown", alertstate="pending", instance="notes-api:8080", job="notes-api", severity="critical", team="notes"} => 1 @[1794855560.037]

ALERTS is a time series that Prometheus writes for every active alert. The alertstate label tells you where it stands. The labels of the rule are already attached. Wait another minute and query again:

promtool query instant http://localhost:9090 ALERTS

Expected:

ALERTS{alertname="NotesApiDown", alertstate="firing", instance="notes-api:8080", job="notes-api", severity="critical", team="notes"} => 1 @[1794855617.674]

The same information as JSON, with the time since which the condition has held:

curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[0]'

Expected (shortened):

{
  "labels": {
    "alertname": "NotesApiDown",
    "instance": "notes-api:8080",
    "job": "notes-api",
    "severity": "critical",
    "team": "notes"
  },
  "state": "firing",
  "activeAt": "2026-11-16T18:59:13.78407747Z",
  …
}

The env label is still missing here. Prometheus only attaches external_labels when sending. Start notes-api again:

docker compose start notes-api

After the next scrape, the alert disappears. curl -s localhost:9090/api/v1/alerts | jq '.data.alerts | length' prints 0.

Tried it: If pending is still there after more than a minute, do the math. for counts from activeAt, and Prometheus only checks every 15 seconds. In the test, the alert became firing 64 seconds after activeAt.

Save your progress? Sign up for free

Drill 6: Fault Instead of Outage

The error rate does not need an outage, only bad responses:

NOTES_ERROR_RATE=0.3 docker compose up -d notes-api

After a little over two minutes:

curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[0].annotations'

Expected:

{
  "summary": "31.28% der Anfragen an /notes scheitern"
}

The value comes from $value and changes with every evaluation. Back to the normal state:

docker compose up -d notes-api

Save your progress? Sign up for free

Checkpoint

promtool check rules prometheus/rules/notes.yml

Expected: SUCCESS: 3 rules found.

curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'

Expected: NotesApiDown, NotesDbDown and NotesHighErrorRate, one line each.

promtool query instant http://localhost:9090 notes_db_up

Expected: one time series with => 1.

Challenge

Write a fourth rule NotesSlow (severity warning, team notes): it fires when 95 % of the requests to /notes take longer than 250 milliseconds, for one minute. Trigger it with NOTES_DELAY.

Note: notes-api provides the histogram notes_http_request_duration_seconds_bucket. histogram_quantile(0.95, …) needs the rates per bucket, aggregated by (le). Reload with curl -X POST localhost:9090/-/reload.

Solution

cat >> prometheus/rules/notes.yml <<'EOF'
      - alert: NotesSlow
        expr: |
          histogram_quantile(0.95,
            sum by (le) (rate(notes_http_request_duration_seconds_bucket{path="/notes"}[2m]))
          ) > 0.25
        for: 1m
        labels:
          severity: warning
          team: notes
        annotations:
          summary: "95 % der Anfragen an /notes dauern bis zu {{ $value | humanizeDuration }}"
EOF
promtool check rules prometheus/rules/notes.yml
curl -s -X POST localhost:9090/-/reload
NOTES_DELAY=300ms docker compose up -d notes-api

Expected: SUCCESS: 4 rules found. After about two minutes:

curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[].annotations.summary'

Expected:

95 % der Anfragen an /notes dauern bis zu 487.5ms

The requests take 300 milliseconds, the alert reports 487.5. Prometheus only knows the bucket boundaries, here 0.25 and 0.5 seconds, and interpolates linearly in between. That is good enough for the threshold, but rough for the display. Back to the normal state:

docker compose up -d notes-api

The NotesSlow rule stays in the file for the next days.

Save your progress? Sign up for free

Enterprise Note

  • Every alert needs a runbook_url. Whoever is woken up at night should not have to search for what to do first.
  • Rules belong in Git and through promtool check rules in CI. For rules with logic, there are unit tests: promtool test rules.
  • Alert on symptoms that users feel (error rate, latency), not on every cause. Cause alerts like NotesDbDown help with diagnosis and get a quieter voice on Day 5.
  • for is a compromise: too short, and every outlier wakes someone up. Too long, and users notice the outage before you do.