Appendix B: Final Exam
Goal: Show without instructions that you have mastered everyday observability: an alert, a dashboard as code, an alert from events and a change to the Collector. · Duration: approx. 90 min · Prerequisite: Day 15
Rules of Play
- Every exercise only describes the goal. You find the way yourself; the days in parentheses help you look things up.
- Every exercise ends with a check block: individual commands with what they must show. Only when all of them match does the exercise count.
- All files live in
~/obs-lab.
cd ~/obs-labExercise 1: Nobody Writes Anymore
Write an alert NotesApiNoWrites (severity
warning, namespace app): it fires when
notes-api hasn’t created a single note for three minutes. Trigger it by
stopping the load generator. (Day 3, Day 4)
Check block (after
kubectl -n app scale deploy/last --replicas 0 and five
minutes):
amtool alert query alertname=NotesApiNoWritesExpected:
Alertname Starts At Summary State
NotesApiNoWrites 2026-11-05 13:12:30 UTC notes-api legt keine Notizen an active
kubectl -n app scale deploy/last --replicas 1After that, the alert must go out within five minutes.
Solution
cat > notes-nowrites.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: notes-api-nowrites
namespace: app
spec:
groups:
- name: notes-api.nowrites
rules:
- alert: NotesApiNoWrites
expr: sum(rate(notes_created_total[2m])) == 0
for: 3m
labels:
severity: warning
namespace: app
annotations:
summary: notes-api legt keine Notizen an
EOF
kubectl apply -f notes-nowrites.yamlrate(...) == 0 instead of absent():
notes-api keeps running and delivers the counter, it just doesn’t grow
anymore. notes_created_total only counts successful notes –
a broken database would trigger the alert too.
Save your progress? Sign up for free
Exercise 2: Logs in the Dashboard
Create a second dashboard as code: uid notes-api-logs,
title notes-api Logs, a panel Logzeilen pro Sekunde
nach Level (time series, data source Loki) and a panel
Fehlerzeilen (logs visualization, only
level ERROR). It goes into the same folder as
the dashboard from Day 6 via Kustomize. (Day 6, Day 8)
Check block (one minute after
kubectl apply -k):
curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/dashboards/uid/notes-api-logs \
| jq -c '[.dashboard.panels[] | {type, ds: .datasource.uid}]'Expected:
[{"type":"timeseries","ds":"loki"},{"type":"logs","ds":"loki"}]
kubectl -n app get configmap -l grafana_dashboard=1 -o nameExpected:
configmap/notes-api-dashboard
configmap/notes-api-logs-dashboard
Solution
cd ~/obs-lab/dashboards
cat > notes-api-logs.json <<'EOF'
{
"uid": "notes-api-logs",
"title": "notes-api Logs",
"tags": ["notes-api"],
"time": {"from": "now-30m", "to": "now"},
"schemaVersion": 41,
"panels": [
{
"id": 1, "type": "timeseries", "title": "Logzeilen pro Sekunde nach Level",
"gridPos": {"x": 0, "y": 0, "w": 24, "h": 7},
"datasource": {"type": "loki", "uid": "loki"},
"targets": [{"refId": "A", "legendFormat": "{{level}}",
"expr": "sum by (level) (rate({app=\"notes-api\"} | json [1m]))"}]
},
{
"id": 2, "type": "logs", "title": "Fehlerzeilen",
"gridPos": {"x": 0, "y": 7, "w": 24, "h": 10},
"datasource": {"type": "loki", "uid": "loki"},
"targets": [{"refId": "A",
"expr": "{app=\"notes-api\"} | json | level=\"ERROR\""}]
}
]
}
EOF
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: app
configMapGenerator:
- name: notes-api-dashboard
files:
- notes-api.json
options:
labels:
grafana_dashboard: "1"
disableNameSuffixHash: true
- name: notes-api-logs-dashboard
files:
- notes-api-logs.json
options:
labels:
grafana_dashboard: "1"
disableNameSuffixHash: true
EOF
kubectl apply -k .
cd ~/obs-labsum by (level) needs | json before it –
only the parser turns level into a label.
Save your progress? Sign up for free
Exercise 3: Alert from Kubernetes Events
Write a log rule PodBackOff (severity
warning): it fires as soon as an event with
reason=BackOff has appeared in the namespace
app within the last five minutes. Trigger it with the
memory eater from Day 13. (Day 8, Day 13)
Check block (three minutes after
kubectl apply -f speicherfresser.yaml):
amtool alert query alertname=PodBackOffExpected:
Alertname Starts At Summary State
PodBackOff 2026-11-05 13:31:20 UTC Pod in app startet immer wieder active
kubectl -n app delete pod speicherfresserSolution
The rule goes into the ConfigMap from Day 8 as a second file – or
into a ConfigMap of its own with the label loki_rule:
cat > events-rules.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: events-log-rules
namespace: monitoring
labels:
loki_rule: "1"
data:
events.yaml: |
groups:
- name: kubernetes.events
rules:
- alert: PodBackOff
expr: |
sum by (namespace) (count_over_time(
{job="kubernetes-events", namespace="app"}
|= "reason=BackOff" [5m])) > 0
labels:
severity: warning
annotations:
summary: Pod in {{ $labels.namespace }} startet immer wieder
EOF
kubectl apply -f events-rules.yaml
kubectl apply -f speicherfresser.yamlThe sidecar puts every file from every ConfigMap into
/rules/fake. Two ConfigMaps, two files, one ruler.
Save your progress? Sign up for free
Exercise 4: Environment on Every Trace
Every trace that goes through the Collector should carry the resource
attribute deployment.environment.name with the value
lab – without changing notes-api. (Day 9, Day 10)
Check block (one minute after the
helm upgrade):
curl -s -G localhost:30320/api/search --data-urlencode limit=1 \
--data-urlencode 'q={ resource.deployment.environment.name = "lab" }' \
| jq -r '.traces[0].rootServiceName'Expected:
notes-api
Solution
A resource processor with action: upsert in
the traces pipeline. Here is the whole
otel.yaml:
cat > otel.yaml <<'EOF'
mode: deployment
fullnameOverride: otel-collector
image:
repository: otel/opentelemetry-collector-k8s
presets:
kubernetesAttributes:
enabled: true
resources:
requests: {cpu: 20m, memory: 64Mi}
limits: {memory: 256Mi}
ports:
jaeger-compact: {enabled: false}
jaeger-thrift: {enabled: false}
jaeger-grpc: {enabled: false}
zipkin: {enabled: false}
metrics: {enabled: true}
serviceMonitor:
enabled: true
config:
receivers:
jaeger: null
zipkin: null
prometheus: null
processors:
resource/env:
attributes:
- key: deployment.environment.name
value: lab
action: upsert
exporters:
debug:
verbosity: basic
otlp_grpc/tempo:
endpoint: tempo:4317
tls:
insecure: true
otlp_http/loki:
endpoint: http://loki:3100/otlp
otlp_http/prometheus:
endpoint: http://kps-prometheus:9090/api/v1/otlp
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, resource/env, batch]
exporters: [debug, otlp_grpc/tempo]
logs:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [otlp_http/loki]
metrics:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [otlp_http/prometheus]
EOF
helm upgrade otel open-telemetry/opentelemetry-collector --version 0.178.2 \
-n monitoring -f otel.yaml --waitresourcechanges resource attributes,attributes(a different processor) the attributes of individual spans.- The order matters:
batchalways comes last, so that what’s finished gets bundled.
Save your progress? Sign up for free
You made it
All four check blocks green? Then you can build, extend and operate an observability stack on Kubernetes – and, more importantly, use it when things are on fire.
Cleaning up is easy: multipass delete --purge obs on the
laptop.