Skip to content
Hands-on IT IT Drills
Cover: Alertmanager and karma hands-on

Training material

Alertmanager and karma hands-on

Alerts, silences and dashboards

From the first alert to the control room. You write Prometheus rules for notes-api, build a routing tree with teams and on-call, shape mails with templates, suppress follow-up alerts with inhibition, silence maintenance windows and run three Alertmanagers in an HA cluster. Then karma brings two environments onto one screen, with filters, multi-grid, silences and acknowledgements, login via Caddy, silence ACLs and healthchecks on the watchdog. Finally everything runs on k3s with kube-prometheus-stack. Every failure in the book really happened during testing.

  • Alertmanager
  • karma
  • Prometheus
  • Observability
  • Kubernetes

Training plan

  1. Foreword
  2. Day 0: The Training Ground
  3. Day 1: Alerts Are Born
  4. Day 2: Alertmanager
  5. Day 3: The Routing Tree
  6. Day 4: Receivers and Templates
  7. Day 5: Creating Quiet
  8. Day 6: High Availability
  9. Day 7: karma
  10. Day 8: Filters and View
  11. Day 9: Multiple Alertmanagers
  12. Day 10: Silences in karma
  13. Day 11: Access
  14. Day 12: History, Health Checks and Troubleshooting
  15. Day 13: karma on Kubernetes
  16. Day 14: Operations
  17. Day 15: Competition
  18. Appendix A: Cheat Sheet
  19. Appendix B: Final Exam