Day 11: Access
Goal: Only logged-in users see karma, every silence carries the real name, and rules determine who may mute what. dev is read-only. · Duration: approx. 60 min · Prerequisite: Day 10
Briefing
- Authentication: karma knows two ways. Either it checks user and password itself (Basic Auth, according to the docs only for testing), or a proxy in front of it takes care of that and passes the name on in a header.
- When someone is logged in, karma sets their name as the author of every silence. Nobody writes “karma” or someone else’s name in there anymore.
- Authorization: groups (
admins,notes) and silence ACLs. The ACLs are rules that are checked in order. The first rule that allows or blocks decides.requireMatcheronly blocks if a matcher is missing, and otherwise lets the next rule check. - ACLs only apply to upstreams with
proxy: true. Only then does every silence go through karma. readonly: truelocks silences for an upstream completely.
Browser ─► Caddy :8000 ── password ok? ── X-Auth-User: bob ─► karma :8080 ─► ACLs ─► Alertmanager
Drill 1: Basic Auth in karma
The quickest protection, directly in
karma/karma.yaml:
cd ~/am-lab
cat >> karma/karma.yaml <<'EOF'
authentication:
basicAuth:
users:
- username: alice
password: lab-karma-pw-123
- username: bob
password: lab-karma-pw-123
EOF
docker compose restart karmacurl -s -o /dev/null -w '%{http_code}\n' localhost:8000/
curl -s -o /dev/null -w '%{http_code}\n' -u alice:lab-karma-pw-123 localhost:8000/
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/healthExpected:
401
200
200
/health and /metrics stay reachable without
login so that health checks and Prometheus keep working. The passwords
are in plain text in the file. That is why the karma docs recommend
Basic Auth only for testing.
Now a silence as bob, with the wrong author in the JSON:
N=$(date -u +%Y-%m-%dT%H:%M:%SZ)
E=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
cat > silence.json <<EOF
{
"matchers": [
{"name": "alertname", "value": "PlatteVoll", "isRegex": false, "isEqual": true},
{"name": "instance", "value": "b", "isRegex": false, "isEqual": true}
],
"startsAt": "$N",
"endsAt": "$E",
"createdBy": "jemand-anderes",
"comment": "Platte b wird morgen getauscht, NOTES-43"
}
EOF
curl -s -u bob:lab-karma-pw-123 -X POST -H 'Content-Type: application/json' \
-d @silence.json localhost:8000/proxy/alertmanager/prod-1/api/v2/silences
amtool silence query instance=bExpected:
{"silenceID":"6828e995-2fcc-4f4e-ae99-082dfd3a3e7d"}
ID Matchers Ends At Created By Comment
6828e995-2fcc-4f4e-ae99-082dfd3a3e7d alertname="PlatteVoll" instance="b" 2026-11-16 21:50:38 UTC bob Platte b wird morgen getauscht, NOTES-43
Created By is bob. karma overwrote
jemand-anderes. Expire the silence again:
amtool silence expire $(amtool silence query instance=b -q)Save your progress? Sign up for free
Drill 2: Groups and ACLs
alice is an admin, bob belongs to team notes. The groups go in
karma.yaml, the rules in a separate file:
cat >> karma/karma.yaml <<'EOF'
authorization:
acl:
silences: /etc/karma/acls.yaml
groups:
- name: admins
members: [alice]
- name: notes
members: [bob]
EOF
cat > karma/acls.yaml <<'EOF'
rules:
- action: block
reason: Regex-Silences sind gesperrt, nimm konkrete Werte
scope:
filters:
- name_re: .+
value_re: .+
isRegex: true
- action: allow
reason: Admins dürfen alles
scope:
groups: [admins]
- action: requireMatcher
reason: Team notes muss team=notes in jede Silence schreiben
scope:
groups: [notes]
matchers:
required:
- name: team
value: notes
isEqual: true
- action: block
reason: Löschen dürfen nur Admins
scope:
operations: [delete]
EOF
docker compose restart karma
docker compose logs --no-log-prefix --since 10s karma | grep 'ACL'Expected:
level=INFO msg="Reading silence ACL config file" path=/etc/karma/acls.yaml
level=INFO msg="Parsed ACL rules" rules=4
The rules from top to bottom:
- No regex silences, for nobody. With
instance=~"pro.*", every later rule could be bypassed. The karma docs explicitly recommend this rule at the very top. - Admins may do everything.
allowends the check. - Team notes must set the matcher
team="notes". That way bob can only mute alerts of his own team. - Deleting only for admins. alice already passed at
rule 2.
operationsonly exists since karma v0.134. Older versions do not check any ACLs at all when deleting (Day 14).
Put it to the test, first bob without team:
P=localhost:8000/proxy/alertmanager/prod-1/api/v2
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -X POST \
-H 'Content-Type: application/json' -d @silence.json $P/silencesExpected:
silence blocked by ACL rule: Team notes muss team=notes in jede Silence schreiben
400
Now with team. Create a second file for it, with a third
matcher:
cat > silence-team.json <<EOF
{
"matchers": [
{"name": "alertname", "value": "PlatteVoll", "isRegex": false, "isEqual": true},
{"name": "instance", "value": "b", "isRegex": false, "isEqual": true},
{"name": "team", "value": "notes", "isRegex": false, "isEqual": true}
],
"startsAt": "$N",
"endsAt": "$E",
"createdBy": "bob",
"comment": "Platte b wird morgen getauscht, NOTES-43"
}
EOF
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -X POST \
-H 'Content-Type: application/json' -d @silence-team.json $P/silencesExpected:
{"silenceID":"c73448b0-51f5-48d9-a55a-4524ec88a0f3"}
200
And delete. Wait half a minute for that, until karma has fetched the new silence:
ID=$(amtool silence query instance=b -q)
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -X DELETE $P/silence/$ID
curl -s -w ' %{http_code}\n' -u alice:lab-karma-pw-123 -X DELETE $P/silence/$IDExpected:
silence blocked by ACL rule: Löschen dürfen nur Admins
400
200
Tried it: If you delete right after creating, you get
silence '…' not found. When deleting, karma checks against its own copy of the silences, and that copy can be up to 30 seconds old (alertmanager.interval).
Warning: The ACLs only apply to the path through karma. The Alertmanager ports 9093 to 9096 are open in the lab. Anyone who reaches them can create any silence with
amtool. In production, only karma and Prometheus may reach the Alertmanagers.
Save your progress? Sign up for free
Drill 3: Login Through a Proxy
In production, a proxy or an SSO service takes care of login. karma only reads the name from a header. Here Caddy plays the proxy, with Basic Auth and hashed passwords. Caddy generates the hash itself:
docker run --rm caddy:2.11.9 caddy hash-password --plaintext lab-karma-pw-123Expected (your hash differs; every call generates a new one):
$2a$14$tPTS0zo18W8cwYc.BVfweeSHNzfIzGqdSmGiDyyvbEdjlstYcXHba
Put your hash into the file:
mkdir -p caddy
cat > caddy/Caddyfile <<'EOF'
:8000 {
basic_auth {
alice $2a$14$tPTS0zo18W8cwYc.BVfweeSHNzfIzGqdSmGiDyyvbEdjlstYcXHba
bob $2a$14$tPTS0zo18W8cwYc.BVfweeSHNzfIzGqdSmGiDyyvbEdjlstYcXHba
}
reverse_proxy karma:8080 {
header_up X-Auth-User {http.auth.user.id}
}
}
EOFbasic_authchecks user and bcrypt hash. Indenting with tabs is common in Caddyfiles; spaces work too.header_up X-Auth-User {http.auth.user.id}sets the header to the logged-in name. If the client sends its ownX-Auth-User, it is overwritten.
karma no longer gets its own port. Only Caddy is reachable from outside:
cat > dashboard.yaml <<'EOF'
services:
karma:
image: ghcr.io/prymitive/karma:v0.134
environment:
CONFIG_FILE: /etc/karma/karma.yaml
volumes:
- ./karma:/etc/karma:ro
caddy:
image: caddy:2.11.9
volumes:
- ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro
ports:
- "8000:8000"
EOFIn karma/karma.yaml, replace the
authentication section with the header method. karma allows
only one method:
authentication:
header:
name: X-Auth-User
value_re: ^(.+)$value_re must have exactly one capture group. Whatever
it matches is the user name.
docker compose up -d karma caddy
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/
curl -s -o /dev/null -w '%{http_code}\n' -u bob:lab-karma-pw-123 localhost:8000/
curl -s -o /dev/null -w '%{http_code}\n' -u bob:falsch localhost:8000/Expected:
401
200
401
The attempt to pose as alice:
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -H 'X-Auth-User: alice' \
-X POST -H 'Content-Type: application/json' -d @silence.json $P/silencesExpected:
silence blocked by ACL rule: Team notes muss team=notes in jede Silence schreiben
400
The rule for bob applies, not the one for alice. Caddy replaced the header.
Warning: karma trusts every header. Anyone who reaches karma directly, without Caddy, is an admin with
X-Auth-User: alice. That is why karma no longer has a port. In Kubernetes, a NetworkPolicy belongs in front of it.
Save your progress? Sign up for free
Drill 4: dev Read-Only
Nobody should create silences in dev. Change the upstream
dev in karma/karma.yaml:
proxy: true becomes readonly: true. Both
together are not allowed.
- name: dev
uri: http://alertmanager-dev:9093
external_uri: http://172.16.0.61:9096
timeout: 10s
readonly: truedocker compose restart karma
docker compose logs --no-log-prefix --since 10s karma | grep 'name=dev'
curl -s -w ' %{http_code}\n' -u alice:lab-karma-pw-123 -X POST \
-H 'Content-Type: application/json' -d @silence-team.json \
localhost:8000/proxy/alertmanager/dev/api/v2/silencesExpected:
level=INFO msg="Configured Alertmanager source" name=dev cluster=dev uri=http://alertmanager-dev:9093 proxy=false readonly=true
Not found 404
There is no proxy path for dev anymore, not even for alice. dev is now missing from the form, and the author is fixed to the logged-in name:
From now on, all curl calls to karma need
-u alice:lab-karma-pw-123 or bob’s credentials.
Save your progress? Sign up for free
Checkpoint
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/Expected: 401.
curl -s -u alice:lab-karma-pw-123 localhost:8000/labelNames.json | jq lengthExpected: a number greater than 0. alice sees the data.
docker compose ps karma --format '{{.Ports}}'Expected: 8080/tcp without 0.0.0.0. karma
is not reachable from outside.
Challenge
Team notes should not mute alerts with
severity="critical". Those are reserved for the on-call
team (alice). Add an ACL rule and test it with bob and alice.
Note: The order decides. The new rule must come
after the admin rule, otherwise it also hits alice.
requireMatcher only decides when a matcher is missing;
otherwise the check continues. A filter compares against
the matchers of the silence.
Solution
In karma/acls.yaml directly after the admin rule:
- action: block
reason: Kritische Alarme schaltet nur die Bereitschaft stumm
scope:
groups: [notes]
filters:
- name: severity
value: criticalA silence with severity="critical". Set the times and
the path again if you have a new shell by now:
N=$(date -u +%Y-%m-%dT%H:%M:%SZ)
E=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
P=localhost:8000/proxy/alertmanager/prod-1/api/v2
cat > silence-critical.json <<EOF
{
"matchers": [
{"name": "alertname", "value": "NotesApiDown", "isRegex": false, "isEqual": true},
{"name": "severity", "value": "critical", "isRegex": false, "isEqual": true},
{"name": "team", "value": "notes", "isRegex": false, "isEqual": true}
],
"startsAt": "$N",
"endsAt": "$E",
"createdBy": "bob",
"comment": "Test der ACL"
}
EOF
docker compose restart karma
curl -s -w ' %{http_code}\n' -u bob:lab-karma-pw-123 -X POST \
-H 'Content-Type: application/json' -d @silence-critical.json $P/silences
curl -s -w ' %{http_code}\n' -u alice:lab-karma-pw-123 -X POST \
-H 'Content-Type: application/json' -d @silence-critical.json $P/silencesExpected:
silence blocked by ACL rule: Kritische Alarme schaltet nur die Bereitschaft stumm
400
{"silenceID":"c3fde3ed-f536-4d80-b7a6-dde71502beea"}
200
Expire alice’s test silence again:
amtool silence expire $(amtool silence query severity=critical -q)Save your progress? Sign up for free
Enterprise Note
- In production, an SSO proxy sits in front of karma (oauth2-proxy,
Cloudflare Access, Keycloak Gatekeeper). It sets user and groups as
headers. karma reads groups with
header.group_nameandgroup_value_re. - karma must only be reachable through this proxy. An open port with header auth is admin access for everyone.
- The Alertmanagers belong behind a network boundary. Otherwise anyone
can bypass the ACLs with
amtool. - Treat ACL files like code: in Git, with review. The first rule blocks regex silences, and the last rule is a deliberate decision for everything that is left.