Skip to content
Hands-on IT IT Drills

Day 14: Operations

Goal: You update karma and the Alertmanagers without downtime, know which configuration wins, recognize the typical failure patterns and can judge when karma is the right tool. · Duration: approx. 50 min · Prerequisite: Day 13

Briefing

  • karma releases a new version about once a month. Releases are tags like v0.134. latest is the main branch and not for production.
  • New versions bring new fields. A configuration that needs a new version breaks when you roll back. --check-config with the target image shows that beforehand.
  • You update Alertmanagers in a cluster one at a time, node by node. The rest of the cluster keeps silences and nflog.
  • Configuration comes from file, environment and flags. The order of precedence is not the one you would expect.

Drill 1: Back to the Compose Lab

k3s has done its job. Stop it and start the Compose project again:

sudo systemctl disable --now k3s
sudo /usr/local/bin/k3s-killall.sh
cd ~/am-lab
docker compose start
docker compose ps --format '{{.Service}} {{.Status}}' | sort

Expected:

alarm-log Up 32 seconds
alertmanager Up 31 seconds
alertmanager-2 Up 30 seconds
alertmanager-3 Up 29 seconds
alertmanager-dev Up 29 seconds
caddy Up 28 seconds
karma Up 27 seconds
last Up 21 seconds
last-dev Up 26 seconds
mailpit Up 26 seconds (healthy)
node-exporter Up 32 seconds
notes-api Up 21 seconds (healthy)
notes-api-dev Up 23 seconds (healthy)
postgres Up 33 seconds (healthy)
prometheus Up 25 seconds
prometheus-dev Up 24 seconds

disable --now stops k3s and prevents it from coming back after a VM reboot. k3s-killall.sh also ends the k3s containers that the service leaves behind. Everything comes back with the state of Day 12, including notes-api, which you had stopped in Day 10.

Save your progress? Sign up for free

Drill 2: Update karma

Which version is current, and what do the release notes say?

curl -s https://api.github.com/repos/prymitive/karma/releases/latest \
  | jq -r '.tag_name, .published_at'
curl -s https://api.github.com/repos/prymitive/karma/releases/tags/v0.134 \
  | jq -r .body | head -8

Expected:

v0.134
2026-10-22T14:41:03Z
### Fixed

- Silence ACL rules are now enforced when deleting silences via the proxy - #7170.
- Silence matcher sliders and label value suggestions now update correctly when …

### Added

- Silence ACL rules can now be scoped to specific operations (`create`, `edit`, `delete`), see [ACLs.md](docs/ACLs.md) - #7170.

Read the release notes before every upgrade. Here there is something important: only since v0.134 do the ACLs also apply when deleting through the proxy. With an older version, bob would have been allowed to expire any silence in Day 11.

The safe procedure for every update: first check the configuration with the new image, then switch over.

docker run --rm -v $PWD/karma:/etc/karma:ro ghcr.io/prymitive/karma:v0.134 \
  --config.file /etc/karma/karma.yaml --check-config 2>&1 | tail -1

Expected:

level=INFO msg="Configuration is valid"

Then change the tag in dashboard.yaml, run docker compose up -d karma and docker compose restart caddy (see below), done. You are already running v0.134. You practice what a switch feels like in the Challenge, with a step back.

Tried it: After every recreation of the karma container, Caddy served the responses of notes-api for a while in the test. curl -u alice:… localhost:8000/version printed 0.8.0, the version of notes-api. Caddy kept an open connection to karma’s old IP, and Docker had meanwhile assigned that IP to notes-api. A docker compose restart caddy after every recreation of karma fixes this. In Kubernetes this does not happen; there a Service always points to current pods.

Save your progress? Sign up for free

Drill 3: Who Wins?

karma reads the same setting from the file, from environment variables and from flags. An experiment with log.level. Create a copy of the configuration in which the file explicitly sets info:

mkdir -p /tmp/kt
cp karma/* /tmp/kt/
sed -i 's/^log:$/log:\n  level: info/' /tmp/kt/karma.yaml
grep -A2 '^log:' /tmp/kt/karma.yaml

Expected:

log:
  level: info
  config: true

Now debug via environment variable and via flag. grep -c DEBUG counts the debug lines:

docker run --rm -e LOG_LEVEL=debug -v /tmp/kt:/etc/karma:ro \
  ghcr.io/prymitive/karma:v0.134 --config.file /etc/karma/karma.yaml \
  --check-config 2>&1 | grep -c DEBUG
docker run --rm -v /tmp/kt:/etc/karma:ro ghcr.io/prymitive/karma:v0.134 \
  --config.file /etc/karma/karma.yaml --log.level=debug \
  --check-config 2>&1 | grep -c DEBUG

Expected:

0
23
Source Example Precedence
Flag --log.level=debug always wins
File log.level: info beats the environment
Environment LOG_LEVEL=debug only if the file says nothing

The environment variable only applies to values that are not in the file. Anyone who wants to override a setting in Kubernetes via env that is already in the ConfigMap will be surprised. That is what the flag is for.

rm -r /tmp/kt

Save your progress? Sign up for free

Drill 4: Rolling Alertmanager Update

An Alertmanager update is a recreation, node by node. As a test, you recreate all three one after the other, without a version change. First count the silences:

amtool silence query | wc -l

Expected: a number, for example 3 (header line plus silences). If all silences have expired, create a test silence with --duration=1h first.

Node 3 first:

docker compose up -d --force-recreate --no-deps alertmanager-3

After 20 seconds:

amtool cluster show -o extended --alertmanager.url=http://localhost:9094 | grep -c ':9094'

Expected: 3. Node 3 is back in the cluster. Only then does the next one follow:

docker compose up -d --force-recreate --no-deps alertmanager-2

Wait 20 seconds again, check for 3 again. Then node 1:

docker compose up -d --force-recreate --no-deps alertmanager

After 20 seconds, again amtool cluster show -o extended --alertmanager.url=http://localhost:9094 | grep -c ':9094', expected 3. Finally:

amtool silence query --alertmanager.url=http://localhost:9095 | wc -l

Expected: the same number as before. The volume is preserved with --force-recreate. Even without a volume, everything would come back from the cluster.

  • --no-deps restarts only this one service.
  • Never two nodes at the same time. A cluster of one node keeps sending, but a failure during that time takes everything down.

Save your progress? Sign up for free

Drill 5: Failure Patterns

What went wrong in this book, and how you recognize it:

Symptom Cause Fix
Notification three times Alertmanager without cluster (Day 6) set --cluster.peer
Alert in karma only on one node sender only sends to one deliver to all nodes
Fallback receiver gets nothing continue route without matchers at the top catch-all route at the end
Watchdog every 2 instead of 1 minute group_interval > repeat_interval own group_interval
Mail without all-clear send_resolved off for mail send_resolved: true
Consequence alert after outage source resolves before the target keep_firing_for on the source
karma does not start unknown field --check-config
karma shows upstream as down Watchdog missing (health check) check Prometheus
Deleting silence: “not found” karma does not know it yet wait 30 seconds
amtool alert add --end starts in the future --end without --start set both
Links in alerts point to container --web.external-url missing set the address
Proxy serves wrong service old connection after recreation restart the proxy

Save your progress? Sign up for free

Drill 6: When karma, When Something Else?

Tool Strength Limit
Alertmanager UI always there, silences one Alertmanager, little overview
karma many Alertmanagers, dedup, ACLs read-only plus silences
Grafana Alerting alerts and dashboards in one its own Alertmanager replacement
On-call service escalation, schedules costs money, alerts come from outside

karma is the right choice when several Alertmanagers or environments should go on one screen, when teams should create silences with rules, and when a control room needs a calm, filterable overview. For escalation and schedules, it does not replace an on-call service. The two complement each other: Alertmanager notifies, karma shows the context.

Save your progress? Sign up for free

Checkpoint

curl -s -u alice:lab-karma-pw-123 localhost:8000/version | jq -r .version

Expected: v0.134. If you get 0.8.0, restart Caddy.

amtool cluster show

Expected: Cluster Status: ready.

docker run --rm -v $PWD/karma:/etc/karma:ro ghcr.io/prymitive/karma:v0.134 \
  --config.file /etc/karma/karma.yaml --check-config 2>&1 | tail -1

Expected: Configuration is valid.

Challenge

After a bug in v0.134, your team wants to go back to v0.133. Check beforehand whether the configuration works with v0.133, and find the spot that does not fit.

Note: Check as in Drill 2, only with the old image. The release notes of v0.134 name what was new.

Solution

docker run --rm -v $PWD/karma:/etc/karma:ro ghcr.io/prymitive/karma:v0.133 \
  --config.file /etc/karma/karma.yaml --check-config 2>&1 | tail -1

Expected (wrapped):

level=ERROR msg="Execution failed" error="failed to parse silence ACL configuration
file \"/etc/karma/acls.yaml\": yaml: unmarshal errors:\n  line 32: field operations
not found in type config.SilenceACLRuleScope"

operations in the last ACL rule only exists since v0.134. A rollback would mean removing the rule that only admins may delete. And with v0.133, ACLs do not apply to deleting anyway. The team is better off staying on v0.134 and reporting the bug.

Had you switched without checking, karma would not have started. With docker compose up -d karma on v0.133, exactly this message appeared in the log in the test, and the container exited.

Save your progress? Sign up for free

Enterprise Note

  • Image tags in Git, updates via Renovate or Dependabot. The pull request runs --check-config with the new image in CI.
  • Alertmanager updates according to the release notes: the API is not stable across all versions. karma only tests certain versions (internal/mock/Makefile in the repo).
  • karma has no state. A new karma only needs its configuration. Backups matter for the Alertmanagers: amtool silence query -o json.
  • A runbook with the failure table from Drill 5 saves time at night.