Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

Everything important from Day 0–15 on a few pages. The number in parentheses is the day on which you practiced it. When in doubt, --help and the documentation at prometheus.io/docs, grafana.com/docs and opentelemetry.io/docs take precedence.

Addresses in the Lab

Service Address on the VM Day
notes-api localhost:30080 0
Prometheus localhost:30090 2
Alertmanager localhost:30093 4
Grafana localhost:30300, admin / lab-grafana-pw-123 5
Loki localhost:30310 (LOKI_ADDR) 7
Tempo localhost:30320 10
Collector otel-collector.monitoring:4317 (gRPC), :4318 (HTTP) 9

Tools

Command What it does Day
promtool query instant <url> '<PromQL>' query at the current point in time 2
amtool alert / alert query name=value active alerts, filtered 4
amtool alert query --silenced … silenced alerts 4
amtool silence add name=value --duration 30m --comment … create a silence 4
amtool silence query -q / expire <ID> IDs / end a silence 4
amtool config routes routing tree 4
logcli labels / labels <name> labels / their values 7
logcli query --quiet --limit N -o raw --since 5m '<LogQL>' lines 7
logcli instant-query --quiet '<LogQL>' number from logs 7
logcli series --analyze-labels '{…}' cardinality 8
curl -G :30320/api/search --data-urlencode 'q=<TraceQL>' search traces 10
curl :30320/api/v2/traces/<ID> fetch a trace 10
curl -u admin:… :30300/api/datasources data sources in Grafana 5
curl -u admin:… ':30300/api/search?query=…' search dashboards 5
kubectl -n app set env deploy/notes-api X=value / X- set / delete 1
kubectl -n app port-forward <pod> 9081:8080 & talk to a pod directly 1
helm history <rel> / helm rollback <rel> <rev> revisions / roll back 14

Faults in notes-api

Variable Effect Day
NOTES_DELAY=300ms every request to /notes waits 1
NOTES_ERROR_RATE=0.2 20 % of requests to /notes end with 500 4
NOTES_LOG_FORMAT=json logs as JSON instead of text 8
NOTES_UPSTREAM=https://… target of /upstream 15
OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME destination and name of the traces 10

PromQL

Expression Meaning Day
metric{label="x", code=~"5.."} selector, =~ regular expression 3
metric[5m] range vector: all values of the last 5 minutes 3
rate(counter[5m]) increase per second, robust against restarts 3
increase(counter[1h]) increase in the window 3
sum by (method) (…) / without (pod) aggregate 3
histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m]))) percentile 3
a / b, a > 0.05 calculate, filter 3, 4
absent(up{job="x"} == 1) 1 if nothing is there 4
(… ) or vector(0) 0 instead of empty 12
topk(3, …) the three largest 13
max by (pod) (a) / max by (pod) (b) pair two sources 13

Recording rules are named level:metric:operation, for example job:notes_http_requests:rate5m (3).

LogQL

Expression Meaning Day
{app="notes-api", namespace="app"} stream selector, at least one = 7
\|= "text", != "text", \|~ "regex" line filter 7
\| logfmt, \| json fields to labels 7, 8
\| status >= 500, \| level="ERROR" filter by fields 7, 8
\| line_format "{{.method}} {{.path}}" rebuild the line 8
rate({…}[5m]), count_over_time({…}[5m]) lines per second / in the window 7
avg_over_time({…} \| json \| unwrap duration_ms [5m]) value of a field 8
absent_over_time({…}[2m]) 1 if no line came 15

TraceQL

Expression Meaning Day
{ resource.service.name = "notes-api" } resource attribute 10
{ span.http.response.status_code >= 500 } span attribute 10
{ duration > 150ms && … } conditions on the same span 10
{ name = "GET /upstream" } intrinsic: name 10
{ status = error && kind = client } intrinsic: status, kind 15
{ … } \| select(resource.k8s.pod.name) attributes into the result 15

Operator Resources

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor          # Day 2, 14
spec:
  selector: {matchLabels: {app.kubernetes.io/name: notes-api}}
  sampleLimit: 500
  endpoints:
    - port: http              # name, not number
      interval: 15s
      metricRelabelings:
        - {sourceLabels: [__name__], regex: go_gc_.*, action: drop}
---
kind: PrometheusRule          # Day 3, 4, 12
spec:
  groups:
    - name: beispiel
      rules:
        - record: job:x:rate5m
          expr: sum by (job) (rate(x_total[5m]))
        - alert: XKaputt
          expr: job:x:rate5m > 1
          for: 2m
          labels: {severity: warning}
          annotations: {summary: "x bei {{ $value }}"}

Sidecars and Labels

ConfigMap with label Who reads it Day
grafana_dashboard: "1" Grafana: dashboard JSON 6
grafana_datasource: "1" Grafana: provisioning file for data sources 7
loki_rule: "1" Loki: rules for the ruler 8

Values Files in the Lab

File Chart Day
kps.yaml kube-prometheus-stack 2, 4, 9, 11
loki.yaml loki (OCI) 7, 8
alloy.yaml alloy 7, 13
otel.yaml opentelemetry-collector 9, 10
tempo.yaml tempo (OCI) 10

The most important switches in kps.yaml: serviceMonitorSelectorNilUsesHelmValues: false (2), alertmanager.config (4), enableOTLPReceiver (9), enableFeatures: [exemplar-storage] (11).

SLO Math

Quantity Formula Example 99.5 %
Error budget 1 − target 0.005
Burn rate error rate ÷ budget 3.6 % ÷ 0.5 % = 7.2
Pager > 14.4 in 1 h and 5 min error rate > 7.2 %
Ticket > 6 in 6 h and 30 min error rate > 3 %