Training material
Observability hands-on
Metrics, logs and traces on Kubernetes
notes-api runs under load on k3s and can be slowed down or broken at the push of a button. Metrics by hand, Prometheus and PromQL, recording rules, Alertmanager and amtool, Grafana and dashboards as code, Loki and Alloy with LogQL and log alerts, the OpenTelemetry Collector, Tempo and TraceQL, exemplars and trace IDs in logs, SLOs with burn-rate alerts, observing Kubernetes and operating the stack itself.
- Observability
- Prometheus
- Grafana
- OpenTelemetry
- Kubernetes
Training plan
- Foreword
- Day 0: The Training Ground
- Day 1: Reading Metrics by Hand
- Day 2: Prometheus in the Cluster
- Day 3: PromQL and Recording Rules
- Day 4: Alerts with Alertmanager
- Day 5: Grafana
- Day 6: Dashboards as Code
- Day 7: Logs with Loki and Alloy
- Day 8: Structured Logs and Log Alerts
- Day 9: The OpenTelemetry Collector
- Day 10: Traces with Tempo
- Day 11: Connecting Metrics, Logs and Traces
- Day 12: SLOs and Burn Rate Alerts
- Day 13: Observing Kubernetes
- Day 14: Running the Stack
- Day 15: Competition
- Appendix A: Cheat Sheet
- Appendix B: Final Exam