Skip to content
Hands-on IT IT Drills

Day 11: Access

Goal: Only logged-in users see karma, every silence carries the real name, and rules determine who may mute what. dev is read-only. · Duration: approx. 60 min · Prerequisite: Day 10

Briefing

  • Authentication: karma knows two ways. Either it checks user and password itself (Basic Auth, according to the docs only for testing), or a proxy in front of it takes care of that and passes the name on in a header.
  • When someone is logged in, karma sets their name as the author of every silence. Nobody writes “karma” or someone else’s name in there anymore.
  • Authorization: groups (admins, notes) and silence ACLs. The ACLs are rules that are checked in order. The first rule that allows or blocks decides. requireMatcher only blocks if a matcher is missing, and otherwise lets the next rule check.
  • ACLs only apply to upstreams with proxy: true. Only then does every silence go through karma.
  • readonly: true locks silences for an upstream completely.
 Browser ─► Caddy :8000 ── password ok? ── X-Auth-User: bob ─► karma :8080 ─► ACLs ─► Alertmanager

Drill 1: Basic Auth in karma

The quickest protection, directly in karma/karma.yaml:

cd ~/am-lab
cat >> karma/karma.yaml <<'EOF'

authentication:
  basicAuth:
    users:
      - username: alice
        password: lab-karma-pw-123
      - username: bob
        password: lab-karma-pw-123
EOF
docker compose restart karma
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/
curl -s -o /dev/null -w '%{http_code}\n' -u alice:lab-karma-pw-123 localhost:8000/
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/health

Expected:

401
200
200

/health and /metrics stay reachable without login so that health checks and Prometheus keep working. The passwords are in plain text in the file. That is why the karma docs recommend Basic Auth only for testing.

Now a silence as bob, with the wrong author in the JSON:

N=$(date -u +%Y-%m-%dT%H:%M:%SZ)
E=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
cat > silence.json <<EOF
{
  "matchers": [
    {"name": "alertname", "value": "PlatteVoll", "isRegex": false, "isEqual": true},
    {"name": "instance", "value": "b", "isRegex": false, "isEqual": true}
  ],
  "startsAt": "$N",
  "endsAt": "$E",
  "createdBy": "jemand-anderes",
  "comment": "Platte b wird morgen getauscht, NOTES-43"
}
EOF
curl -s -u bob:lab-karma-pw-123 -X POST -H 'Content-Type: application/json' \
  -d @silence.json localhost:8000/proxy/alertmanager/prod-1/api/v2/silences
amtool silence query instance=b

Expected:

{"silenceID":"6828e995-2fcc-4f4e-ae99-082dfd3a3e7d"}
ID                                    Matchers                             Ends At                  Created By  Comment
6828e995-2fcc-4f4e-ae99-082dfd3a3e7d  alertname="PlatteVoll" instance="b"  2026-11-16 21:50:38 UTC  bob         Platte b wird morgen getauscht, NOTES-43

Created By is bob. karma overwrote jemand-anderes. Expire the silence again:

amtool silence expire $(amtool silence query instance=b -q)

Save your progress? Sign up for free

Drill 2: Groups and ACLs

alice is an admin, bob belongs to team notes. The groups go in karma.yaml, the rules in a separate file:

cat >> karma/karma.yaml <<'EOF'

authorization:
  acl:
    silences: /etc/karma/acls.yaml
  groups:
    - name: admins
      members: [alice]
    - name: notes
      members: [bob]
EOF
cat > karma/acls.yaml <<'EOF'
rules:
  - action: block
    reason: Regex-Silences sind gesperrt, nimm konkrete Werte
    scope:
      filters:
        - name_re: .+
          value_re: .+
          isRegex: true
  - action: allow
    reason: Admins dürfen alles
    scope:
      groups: [admins]
  - action: requireMatcher
    reason: Team notes muss team=notes in jede Silence schreiben
    scope:
      groups: [notes]
    matchers:
      required:
        - name: team
          value: notes
          isEqual: true
  - action: block
    reason: Löschen dürfen nur Admins
    scope:
      operations: [delete]
EOF
docker compose restart karma
docker compose logs --no-log-prefix --since 10s karma | grep 'ACL'

Expected:

level=INFO msg="Reading silence ACL config file" path=/etc/karma/acls.yaml
level=INFO msg="Parsed ACL rules" rules=4

The rules from top to bottom:

  1. No regex silences, for nobody. With instance=~"pro.*", every later rule could be bypassed. The karma docs explicitly recommend this rule at the very top.
  2. Admins may do everything. allow ends the check.
  3. Team notes must set the matcher team="notes". That way bob can only mute alerts of his own team.
  4. Deleting only for admins. alice already passed at rule 2. operations only exists since karma v0.134. Older versions do not check any ACLs at all when deleting (Day 14).

Put it to the test, first bob without team:

P=localhost:8000/proxy/alertmanager/prod-1/api/v2
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -X POST \
  -H 'Content-Type: application/json' -d @silence.json $P/silences

Expected:

silence blocked by ACL rule: Team notes muss team=notes in jede Silence schreiben
 400

Now with team. Create a second file for it, with a third matcher:

cat > silence-team.json <<EOF
{
  "matchers": [
    {"name": "alertname", "value": "PlatteVoll", "isRegex": false, "isEqual": true},
    {"name": "instance", "value": "b", "isRegex": false, "isEqual": true},
    {"name": "team", "value": "notes", "isRegex": false, "isEqual": true}
  ],
  "startsAt": "$N",
  "endsAt": "$E",
  "createdBy": "bob",
  "comment": "Platte b wird morgen getauscht, NOTES-43"
}
EOF
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -X POST \
  -H 'Content-Type: application/json' -d @silence-team.json $P/silences

Expected:

{"silenceID":"c73448b0-51f5-48d9-a55a-4524ec88a0f3"}
 200

And delete. Wait half a minute for that, until karma has fetched the new silence:

ID=$(amtool silence query instance=b -q)
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -X DELETE $P/silence/$ID
curl -s -w ' %{http_code}\n' -u alice:lab-karma-pw-123 -X DELETE $P/silence/$ID

Expected:

silence blocked by ACL rule: Löschen dürfen nur Admins
 400
 200

Tried it: If you delete right after creating, you get silence '…' not found. When deleting, karma checks against its own copy of the silences, and that copy can be up to 30 seconds old (alertmanager.interval).

Warning: The ACLs only apply to the path through karma. The Alertmanager ports 9093 to 9096 are open in the lab. Anyone who reaches them can create any silence with amtool. In production, only karma and Prometheus may reach the Alertmanagers.

Save your progress? Sign up for free

Drill 3: Login Through a Proxy

In production, a proxy or an SSO service takes care of login. karma only reads the name from a header. Here Caddy plays the proxy, with Basic Auth and hashed passwords. Caddy generates the hash itself:

docker run --rm caddy:2.11.9 caddy hash-password --plaintext lab-karma-pw-123

Expected (your hash differs; every call generates a new one):

$2a$14$tPTS0zo18W8cwYc.BVfweeSHNzfIzGqdSmGiDyyvbEdjlstYcXHba

Put your hash into the file:

mkdir -p caddy
cat > caddy/Caddyfile <<'EOF'
:8000 {
    basic_auth {
        alice $2a$14$tPTS0zo18W8cwYc.BVfweeSHNzfIzGqdSmGiDyyvbEdjlstYcXHba
        bob $2a$14$tPTS0zo18W8cwYc.BVfweeSHNzfIzGqdSmGiDyyvbEdjlstYcXHba
    }
    reverse_proxy karma:8080 {
        header_up X-Auth-User {http.auth.user.id}
    }
}
EOF
  • basic_auth checks user and bcrypt hash. Indenting with tabs is common in Caddyfiles; spaces work too.
  • header_up X-Auth-User {http.auth.user.id} sets the header to the logged-in name. If the client sends its own X-Auth-User, it is overwritten.

karma no longer gets its own port. Only Caddy is reachable from outside:

cat > dashboard.yaml <<'EOF'
services:
  karma:
    image: ghcr.io/prymitive/karma:v0.134
    environment:
      CONFIG_FILE: /etc/karma/karma.yaml
    volumes:
      - ./karma:/etc/karma:ro

  caddy:
    image: caddy:2.11.9
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro
    ports:
      - "8000:8000"
EOF

In karma/karma.yaml, replace the authentication section with the header method. karma allows only one method:

authentication:
  header:
    name: X-Auth-User
    value_re: ^(.+)$

value_re must have exactly one capture group. Whatever it matches is the user name.

docker compose up -d karma caddy
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/
curl -s -o /dev/null -w '%{http_code}\n' -u bob:lab-karma-pw-123 localhost:8000/
curl -s -o /dev/null -w '%{http_code}\n' -u bob:falsch localhost:8000/

Expected:

401
200
401

The attempt to pose as alice:

curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -H 'X-Auth-User: alice' \
  -X POST -H 'Content-Type: application/json' -d @silence.json $P/silences

Expected:

silence blocked by ACL rule: Team notes muss team=notes in jede Silence schreiben
 400

The rule for bob applies, not the one for alice. Caddy replaced the header.

Warning: karma trusts every header. Anyone who reaches karma directly, without Caddy, is an admin with X-Auth-User: alice. That is why karma no longer has a port. In Kubernetes, a NetworkPolicy belongs in front of it.

Save your progress? Sign up for free

Drill 4: dev Read-Only

Nobody should create silences in dev. Change the upstream dev in karma/karma.yaml: proxy: true becomes readonly: true. Both together are not allowed.

    - name: dev
      uri: http://alertmanager-dev:9093
      external_uri: http://172.16.0.61:9096
      timeout: 10s
      readonly: true
docker compose restart karma
docker compose logs --no-log-prefix --since 10s karma | grep 'name=dev'
curl -s -w ' %{http_code}\n' -u alice:lab-karma-pw-123 -X POST \
  -H 'Content-Type: application/json' -d @silence-team.json \
  localhost:8000/proxy/alertmanager/dev/api/v2/silences

Expected:

level=INFO msg="Configured Alertmanager source" name=dev cluster=dev uri=http://alertmanager-dev:9093 proxy=false readonly=true
Not found 404

There is no proxy path for dev anymore, not even for alice. dev is now missing from the form, and the author is fixed to the logged-in name:

The form for bob: only cluster prod left, author fixed

From now on, all curl calls to karma need -u alice:lab-karma-pw-123 or bob’s credentials.

Save your progress? Sign up for free

Checkpoint

curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/

Expected: 401.

curl -s -u alice:lab-karma-pw-123 localhost:8000/labelNames.json | jq length

Expected: a number greater than 0. alice sees the data.

docker compose ps karma --format '{{.Ports}}'

Expected: 8080/tcp without 0.0.0.0. karma is not reachable from outside.

Challenge

Team notes should not mute alerts with severity="critical". Those are reserved for the on-call team (alice). Add an ACL rule and test it with bob and alice.

Note: The order decides. The new rule must come after the admin rule, otherwise it also hits alice. requireMatcher only decides when a matcher is missing; otherwise the check continues. A filter compares against the matchers of the silence.

Solution

In karma/acls.yaml directly after the admin rule:

  - action: block
    reason: Kritische Alarme schaltet nur die Bereitschaft stumm
    scope:
      groups: [notes]
      filters:
        - name: severity
          value: critical

A silence with severity="critical". Set the times and the path again if you have a new shell by now:

N=$(date -u +%Y-%m-%dT%H:%M:%SZ)
E=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
P=localhost:8000/proxy/alertmanager/prod-1/api/v2
cat > silence-critical.json <<EOF
{
  "matchers": [
    {"name": "alertname", "value": "NotesApiDown", "isRegex": false, "isEqual": true},
    {"name": "severity", "value": "critical", "isRegex": false, "isEqual": true},
    {"name": "team", "value": "notes", "isRegex": false, "isEqual": true}
  ],
  "startsAt": "$N",
  "endsAt": "$E",
  "createdBy": "bob",
  "comment": "Test der ACL"
}
EOF
docker compose restart karma
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -X POST \
  -H 'Content-Type: application/json' -d @silence-critical.json $P/silences
curl -s -w ' %{http_code}\n' -u alice:lab-karma-pw-123 -X POST \
  -H 'Content-Type: application/json' -d @silence-critical.json $P/silences

Expected:

silence blocked by ACL rule: Kritische Alarme schaltet nur die Bereitschaft stumm
 400
{"silenceID":"c3fde3ed-f536-4d80-b7a6-dde71502beea"}
 200

Expire alice’s test silence again:

amtool silence expire $(amtool silence query severity=critical -q)

Save your progress? Sign up for free

Enterprise Note

  • In production, an SSO proxy sits in front of karma (oauth2-proxy, Cloudflare Access, Keycloak Gatekeeper). It sets user and groups as headers. karma reads groups with header.group_name and group_value_re.
  • karma must only be reachable through this proxy. An open port with header auth is admin access for everyone.
  • The Alertmanagers belong behind a network boundary. Otherwise anyone can bypass the ACLs with amtool.
  • Treat ACL files like code: in Git, with review. The first rule blocks regex silences, and the last rule is a deliberate decision for everything that is left.