Skip to content
Hands-on IT IT Drills
Cover: Observability hands-on

Training material

Observability hands-on

Metrics, logs and traces on Kubernetes

notes-api runs under load on k3s and can be slowed down or broken at the push of a button. Metrics by hand, Prometheus and PromQL, recording rules, Alertmanager and amtool, Grafana and dashboards as code, Loki and Alloy with LogQL and log alerts, the OpenTelemetry Collector, Tempo and TraceQL, exemplars and trace IDs in logs, SLOs with burn-rate alerts, observing Kubernetes and operating the stack itself.

  • Observability
  • Prometheus
  • Grafana
  • OpenTelemetry
  • Kubernetes

Training plan

  1. Foreword
  2. Day 0: The Training Ground
  3. Day 1: Reading Metrics by Hand
  4. Day 2: Prometheus in the Cluster
  5. Day 3: PromQL and Recording Rules
  6. Day 4: Alerts with Alertmanager
  7. Day 5: Grafana
  8. Day 6: Dashboards as Code
  9. Day 7: Logs with Loki and Alloy
  10. Day 8: Structured Logs and Log Alerts
  11. Day 9: The OpenTelemetry Collector
  12. Day 10: Traces with Tempo
  13. Day 11: Connecting Metrics, Logs and Traces
  14. Day 12: SLOs and Burn Rate Alerts
  15. Day 13: Observing Kubernetes
  16. Day 14: Running the Stack
  17. Day 15: Competition
  18. Appendix A: Cheat Sheet
  19. Appendix B: Final Exam