Skip to content
Hands-on IT IT Drills

Day 7: karma

Goal: karma runs next to Alertmanager, shows all alerts as a dashboard and reads its settings from a file. You know the UI and the HTTP API behind it. · Duration: approx. 45 min · Prerequisite: Day 6

Briefing

  • karma is a dashboard for Alertmanager, written in Go and React, created at Cloudflare (formerly “unsee”). Project: github.com/prymitive/karma.
  • karma queries the API v2 of one or more Alertmanagers at fixed intervals, merges the alerts and shows them as tiles. It does not change any alerts.
  • karma can write only one thing: create, edit and expire silences. That happens either directly from the browser to Alertmanager or through karma as a proxy.
  • karma is configured via a YAML file, environment variables or flags. For more than one Alertmanager, you need the file.
 Browser ──► karma :8000 ── every 30 s GET /api/v2/alerts/groups, /silences ──► alertmanager :9093
               │
               └── create silence (proxy: true) ── POST /api/v2/silences ──────►

Drill 1: The Quickest Way

One environment variable is enough to start with:

cd ~/am-lab
cat > dashboard.yaml <<'EOF'
services:
  karma:
    image: ghcr.io/prymitive/karma:v0.134
    environment:
      ALERTMANAGER_URI: http://alertmanager:9093
    ports:
      - "8000:8080"
EOF
echo "  - dashboard.yaml" >> compose.yaml
docker compose up -d karma
docker compose logs --no-log-prefix karma

Expected (shortened):

level=INFO msg="Version: v0.134"
level=INFO msg="Configured Alertmanager source" name=default cluster=default uri=http://alertmanager:9093 proxy=false readonly=false
level=INFO msg="Initial Alertmanager collection"
level=INFO msg="Upstream version" version=0.35.0 alertmanager=default
level=INFO msg="Got silences" alertmanager=default silences=3 duration=2.978019ms
level=INFO msg="Collected alert groups" alertmanager=default groups=2 duration=2.379345ms
level=INFO msg="Deduplicating alert groups" alertmanager=default groups=2
…
level=INFO msg="Starting HTTP server" address=[::]:8080
  • karma listens on 8080 inside the container. That port is already taken on the VM by notes-api, so karma is on 8000 outside.
  • Before starting, karma fetches everything once. Only then does it accept requests.
  • The upstream is called default because you did not give it a name.
curl -s localhost:8000/health
curl -s localhost:8000/version

Expected:

Pong
{
  "version": "v0.134",
  "golang": "go1.27.3"
}

Save your progress? Sign up for free

Drill 2: The Configuration File

mkdir -p karma
cat > karma/karma.yaml <<'EOF'
karma:
  name: am-lab

alertmanager:
  interval: 30s
  servers:
    - name: prod
      uri: http://alertmanager:9093
      external_uri: http://172.16.0.61:9093
      timeout: 10s
      proxy: true

log:
  config: true
EOF
  • karma.name appears in the title of the browser tab. With several karma instances, you know which one you are looking at.
  • interval determines how often karma queries all Alertmanagers, at most 15m.
  • uri is used by karma itself, so it is the name in the Compose network. external_uri is used by the browser for links into the Alertmanager UI. Use your VM address.
  • proxy: true: the browser sends silences to karma, and karma passes them on. The browser then does not need to reach Alertmanager. On Day 11, the permission check needs this.
  • log.config: true writes the entire configuration to the log at startup, with all default values.

The Compose building block now reads the file:

cat > dashboard.yaml <<'EOF'
services:
  karma:
    image: ghcr.io/prymitive/karma:v0.134
    environment:
      CONFIG_FILE: /etc/karma/karma.yaml
    volumes:
      - ./karma:/etc/karma:ro
    ports:
      - "8000:8080"
EOF
docker compose up -d karma
docker compose logs --no-log-prefix karma | grep -A9 'msg=alertmanager:'

Expected:

level=INFO msg=alertmanager:
level=INFO msg="  interval: 30s"
level=INFO msg="  servers:"
level=INFO msg="    - cluster: \"\""
level=INFO msg="      name: prod"
level=INFO msg="      uri: http://alertmanager:9093"
level=INFO msg="      external_uri: http://172.16.0.61:9093"
level=INFO msg="      proxy_url: \"\""
level=INFO msg="      timeout: 10s"
level=INFO msg="      proxy: true"

karma has no reload. After every change to the file, you restart the container: docker compose restart karma.

Save your progress? Sign up for free

Drill 3: Something to Look At

So that the dashboard is not empty, you create a few alerts. You already know the error rate and the node-exporter:

NOTES_ERROR_RATE=0.3 docker compose up -d notes-api
docker compose stop node-exporter

Plus test alerts that last three hours:

N=$(date -u +%Y-%m-%dT%H:%M:%SZ)
E=$(date -u -d '+3 hours' +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=a \
  --annotation='summary="Platte a zu 91 % voll"' --start=$N --end=$E
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=b \
  --annotation='summary="Platte b zu 92 % voll"' --start=$N --end=$E
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=c \
  --annotation='summary="Platte c zu 95 % voll"' --start=$N --end=$E
amtool alert add alertname=Werbeaktion team=marketing severity=warning \
  --start=$N --end=$E

Tip: Did you take a break between two days? Test alerts live for three hours, the silences from Day 5 for two and 24 hours. If they are missing, create them again with the same commands, the silences as in Day 5 Drill 3 with amtool silence add. The numbers in the next days will then match again.

Tried it: With --end alone, amtool 0.35 also sets the start to the end. The alert then has startsAt three hours in the future and looks odd in karma. Always pass --start as well.

Prometheus needs a little over two minutes for the error rate. After that, open http://172.16.0.61:8000 in your browser.

karma with one Alertmanager: every tile is an alert group

This is how you read the picture:

  • Every tile is an alert group, the way Alertmanager forms it. The title is made of the labels from group_by, here alertname and env.
  • The numbers on the right of the title: red are active alerts, green suppressed ones. For PlatteVoll, two are active and one is muted.
  • Below each alert are its own labels. Labels that all alerts of a group share move into the footer of the tile.
  • Silences appear directly on the alert, with comment, author and remaining time. The bar below shows how much of the duration has already passed.
  • Labels with @ come from karma: @receiver is the receiver of the group, @state the state, and later @alertmanager the upstream.

Two things stand out. Almost every group appears twice, once with @receiver: archiv and once with the team’s receiver. karma forms groups per receiver, and the archive route from Day 3 gets everything. Also, NodeExporterDown is green for team plattform: the after-hours interval from Day 5 muted the group, and karma counts that as suppressed. If you are reading this on a weekday before 6 p.m. or on the weekend, it is red, and one suppressed alert is missing from the counters in the next days. Day 8 takes care of both.

A click on the number at the top left opens the Overview: the most frequent values of every label across all alerts.

The Overview: which labels occur how often

Here you see right away that the archive accounts for half of all entries. A click on a value sets it as a filter.

Save your progress? Sign up for free

Drill 4: The API Behind the UI

Everything the UI shows, karma also delivers as JSON. For checks from the command line, three endpoints are enough.

The names of all labels currently present:

curl -s localhost:8000/labelNames.json | jq -c .

Expected:

["alertname","env","instance","job","severity","team"]

The counters of the Overview:

curl -s localhost:8000/counters.json | jq '.total'

Expected:

14

The list of alerts, each as a list of its labels:

curl -s localhost:8000/alertList.json | jq -c '.alerts[0]'

Expected:

[{"name":"alertname","value":"NodeExporterDown"},{"name":"env","value":"prod"},{"name":"instance","value":"node-exporter:9100"},{"name":"job","value":"node"},{"name":"severity","value":"warning"},{"name":"team","value":"plattform"}]

The numbers from the API and the UI may differ for a moment. The UI fetches new data every 30 seconds, and karma itself also queries Alertmanager every 30 seconds.

Save your progress? Sign up for free

Checkpoint

curl -s localhost:8000/health

Expected: Pong.

docker compose logs --no-log-prefix karma | grep 'Configured Alertmanager'

Expected: one line with name=prod and proxy=true.

curl -s localhost:8000/labelNames.json | jq length

Expected: 6; karma knows the labels of your alerts.

Challenge

Using only curl, find out which alerts are currently suppressed and how often each name occurs.

Note: counters.json and alertList.json take filters as the parameter q, exactly like the search bar. The filter for the state is called @state=suppressed. curl -G --data-urlencode encodes it correctly.

Solution

curl -s -G localhost:8000/counters.json --data-urlencode 'q=@state=suppressed' \
  | jq -c '.counters[] | select(.name=="alertname") | .values[] | [.value, .hits]'

Expected:

["PlatteVoll",2]
["Werbeaktion",2]
["NodeExporterDown",1]

PlatteVoll (instance a) and Werbeaktion are muted by the silences from Day 5, twice each, because both receivers show them. NodeExporterDown once, only in the group of team plattform with its after-hours interval. In the archive, it is active.

Save your progress? Sign up for free

Enterprise Note

  • karma only needs read access to the Alertmanager API. With proxy: true, the users’ browsers do not need to reach Alertmanager at all. That is the better network boundary.
  • Set external_uri to the address that humans reach, that is the ingress or DNS name, not the internal service name.
  • Pin the image to a version (v0.134), not to latest. For karma, latest is the current state of the main branch, not the latest release.
  • /health always answers with Pong, even if no Alertmanager is reachable. Whether the upstreams are alive is shown by /metrics (Day 12).