Appendix A: Cheat Sheet
Everything important from Day 0–15 on a few pages. The number in
parentheses is the day on which you practiced it. When in doubt,
--help and the documentation at
prometheus.io/docs, grafana.com/docs and
opentelemetry.io/docs take precedence.
Addresses in the Lab
| Service |
Address on the VM |
Day |
| notes-api |
localhost:30080 |
0 |
| Prometheus |
localhost:30090 |
2 |
| Alertmanager |
localhost:30093 |
4 |
| Grafana |
localhost:30300, admin /
lab-grafana-pw-123 |
5 |
| Loki |
localhost:30310 (LOKI_ADDR) |
7 |
| Tempo |
localhost:30320 |
10 |
| Collector |
otel-collector.monitoring:4317 (gRPC),
:4318 (HTTP) |
9 |
Tools
| Command |
What it does |
Day |
promtool query instant <url> '<PromQL>' |
query at the current point in time |
2 |
amtool alert / alert query name=value |
active alerts, filtered |
4 |
amtool alert query --silenced … |
silenced alerts |
4 |
amtool silence add name=value --duration 30m --comment … |
create a silence |
4 |
amtool silence query -q /
expire <ID> |
IDs / end a silence |
4 |
amtool config routes |
routing tree |
4 |
logcli labels / labels <name> |
labels / their values |
7 |
logcli query --quiet --limit N -o raw --since 5m '<LogQL>' |
lines |
7 |
logcli instant-query --quiet '<LogQL>' |
number from logs |
7 |
logcli series --analyze-labels '{…}' |
cardinality |
8 |
curl -G :30320/api/search --data-urlencode 'q=<TraceQL>' |
search traces |
10 |
curl :30320/api/v2/traces/<ID> |
fetch a trace |
10 |
curl -u admin:… :30300/api/datasources |
data sources in Grafana |
5 |
curl -u admin:… ':30300/api/search?query=…' |
search dashboards |
5 |
kubectl -n app set env deploy/notes-api X=value /
X- |
set / delete |
1 |
kubectl -n app port-forward <pod> 9081:8080 & |
talk to a pod directly |
1 |
helm history <rel> /
helm rollback <rel> <rev> |
revisions / roll back |
14 |
Faults in notes-api
| Variable |
Effect |
Day |
NOTES_DELAY=300ms |
every request to /notes waits |
1 |
NOTES_ERROR_RATE=0.2 |
20 % of requests to /notes end with 500 |
4 |
NOTES_LOG_FORMAT=json |
logs as JSON instead of text |
8 |
NOTES_UPSTREAM=https://… |
target of /upstream |
15 |
OTEL_EXPORTER_OTLP_ENDPOINT,
OTEL_SERVICE_NAME |
destination and name of the traces |
10 |
PromQL
| Expression |
Meaning |
Day |
metric{label="x", code=~"5.."} |
selector, =~ regular expression |
3 |
metric[5m] |
range vector: all values of the last 5 minutes |
3 |
rate(counter[5m]) |
increase per second, robust against restarts |
3 |
increase(counter[1h]) |
increase in the window |
3 |
sum by (method) (…) / without (pod) |
aggregate |
3 |
histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m]))) |
percentile |
3 |
a / b, a > 0.05 |
calculate, filter |
3, 4 |
absent(up{job="x"} == 1) |
1 if nothing is there |
4 |
(… ) or vector(0) |
0 instead of empty |
12 |
topk(3, …) |
the three largest |
13 |
max by (pod) (a) / max by (pod) (b) |
pair two sources |
13 |
Recording rules are named level:metric:operation, for
example job:notes_http_requests:rate5m (3).
LogQL
| Expression |
Meaning |
Day |
{app="notes-api", namespace="app"} |
stream selector, at least one = |
7 |
\|= "text", != "text",
\|~ "regex" |
line filter |
7 |
\| logfmt, \| json |
fields to labels |
7, 8 |
\| status >= 500, \| level="ERROR" |
filter by fields |
7, 8 |
\| line_format "{{.method}} {{.path}}" |
rebuild the line |
8 |
rate({…}[5m]),
count_over_time({…}[5m]) |
lines per second / in the window |
7 |
avg_over_time({…} \| json \| unwrap duration_ms [5m]) |
value of a field |
8 |
absent_over_time({…}[2m]) |
1 if no line came |
15 |
TraceQL
| Expression |
Meaning |
Day |
{ resource.service.name = "notes-api" } |
resource attribute |
10 |
{ span.http.response.status_code >= 500 } |
span attribute |
10 |
{ duration > 150ms && … } |
conditions on the same span |
10 |
{ name = "GET /upstream" } |
intrinsic: name |
10 |
{ status = error && kind = client } |
intrinsic: status, kind |
15 |
{ … } \| select(resource.k8s.pod.name) |
attributes into the result |
15 |
Operator Resources
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor # Day 2, 14
spec:
selector: {matchLabels: {app.kubernetes.io/name: notes-api}}
sampleLimit: 500
endpoints:
- port: http # name, not number
interval: 15s
metricRelabelings:
- {sourceLabels: [__name__], regex: go_gc_.*, action: drop}
---
kind: PrometheusRule # Day 3, 4, 12
spec:
groups:
- name: beispiel
rules:
- record: job:x:rate5m
expr: sum by (job) (rate(x_total[5m]))
- alert: XKaputt
expr: job:x:rate5m > 1
for: 2m
labels: {severity: warning}
annotations: {summary: "x bei {{ $value }}"}
Sidecars and Labels
| ConfigMap with label |
Who reads it |
Day |
grafana_dashboard: "1" |
Grafana: dashboard JSON |
6 |
grafana_datasource: "1" |
Grafana: provisioning file for data sources |
7 |
loki_rule: "1" |
Loki: rules for the ruler |
8 |
Values Files in the Lab
| File |
Chart |
Day |
kps.yaml |
kube-prometheus-stack |
2, 4, 9, 11 |
loki.yaml |
loki (OCI) |
7, 8 |
alloy.yaml |
alloy |
7, 13 |
otel.yaml |
opentelemetry-collector |
9, 10 |
tempo.yaml |
tempo (OCI) |
10 |
The most important switches in kps.yaml:
serviceMonitorSelectorNilUsesHelmValues: false (2),
alertmanager.config (4), enableOTLPReceiver
(9), enableFeatures: [exemplar-storage] (11).
SLO Math
| Quantity |
Formula |
Example 99.5 % |
| Error budget |
1 − target |
0.005 |
| Burn rate |
error rate ÷ budget |
3.6 % ÷ 0.5 % = 7.2 |
| Pager |
> 14.4 in 1 h and 5 min |
error rate > 7.2 % |
| Ticket |
> 6 in 6 h and 30 min |
error rate > 3 % |