Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Show without instructions that you have mastered everyday observability: an alert, a dashboard as code, an alert from events and a change to the Collector. · Duration: approx. 90 min · Prerequisite: Day 15

Rules of Play

  • Every exercise only describes the goal. You find the way yourself; the days in parentheses help you look things up.
  • Every exercise ends with a check block: individual commands with what they must show. Only when all of them match does the exercise count.
  • All files live in ~/obs-lab.
cd ~/obs-lab

Exercise 1: Nobody Writes Anymore

Write an alert NotesApiNoWrites (severity warning, namespace app): it fires when notes-api hasn’t created a single note for three minutes. Trigger it by stopping the load generator. (Day 3, Day 4)

Check block (after kubectl -n app scale deploy/last --replicas 0 and five minutes):

amtool alert query alertname=NotesApiNoWrites

Expected:

Alertname         Starts At                Summary                         State
NotesApiNoWrites  2026-11-05 13:12:30 UTC  notes-api legt keine Notizen an  active
kubectl -n app scale deploy/last --replicas 1

After that, the alert must go out within five minutes.

Solution

cat > notes-nowrites.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: notes-api-nowrites
  namespace: app
spec:
  groups:
    - name: notes-api.nowrites
      rules:
        - alert: NotesApiNoWrites
          expr: sum(rate(notes_created_total[2m])) == 0
          for: 3m
          labels:
            severity: warning
            namespace: app
          annotations:
            summary: notes-api legt keine Notizen an
EOF
kubectl apply -f notes-nowrites.yaml

rate(...) == 0 instead of absent(): notes-api keeps running and delivers the counter, it just doesn’t grow anymore. notes_created_total only counts successful notes – a broken database would trigger the alert too.

Save your progress? Sign up for free

Exercise 2: Logs in the Dashboard

Create a second dashboard as code: uid notes-api-logs, title notes-api Logs, a panel Logzeilen pro Sekunde nach Level (time series, data source Loki) and a panel Fehlerzeilen (logs visualization, only level ERROR). It goes into the same folder as the dashboard from Day 6 via Kustomize. (Day 6, Day 8)

Check block (one minute after kubectl apply -k):

curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/dashboards/uid/notes-api-logs \
  | jq -c '[.dashboard.panels[] | {type, ds: .datasource.uid}]'

Expected:

[{"type":"timeseries","ds":"loki"},{"type":"logs","ds":"loki"}]
kubectl -n app get configmap -l grafana_dashboard=1 -o name

Expected:

configmap/notes-api-dashboard
configmap/notes-api-logs-dashboard

Solution

cd ~/obs-lab/dashboards
cat > notes-api-logs.json <<'EOF'
{
  "uid": "notes-api-logs",
  "title": "notes-api Logs",
  "tags": ["notes-api"],
  "time": {"from": "now-30m", "to": "now"},
  "schemaVersion": 41,
  "panels": [
    {
      "id": 1, "type": "timeseries", "title": "Logzeilen pro Sekunde nach Level",
      "gridPos": {"x": 0, "y": 0, "w": 24, "h": 7},
      "datasource": {"type": "loki", "uid": "loki"},
      "targets": [{"refId": "A", "legendFormat": "{{level}}",
        "expr": "sum by (level) (rate({app=\"notes-api\"} | json [1m]))"}]
    },
    {
      "id": 2, "type": "logs", "title": "Fehlerzeilen",
      "gridPos": {"x": 0, "y": 7, "w": 24, "h": 10},
      "datasource": {"type": "loki", "uid": "loki"},
      "targets": [{"refId": "A",
        "expr": "{app=\"notes-api\"} | json | level=\"ERROR\""}]
    }
  ]
}
EOF
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: app
configMapGenerator:
  - name: notes-api-dashboard
    files:
      - notes-api.json
    options:
      labels:
        grafana_dashboard: "1"
      disableNameSuffixHash: true
  - name: notes-api-logs-dashboard
    files:
      - notes-api-logs.json
    options:
      labels:
        grafana_dashboard: "1"
      disableNameSuffixHash: true
EOF
kubectl apply -k .
cd ~/obs-lab

sum by (level) needs | json before it – only the parser turns level into a label.

Save your progress? Sign up for free

Exercise 3: Alert from Kubernetes Events

Write a log rule PodBackOff (severity warning): it fires as soon as an event with reason=BackOff has appeared in the namespace app within the last five minutes. Trigger it with the memory eater from Day 13. (Day 8, Day 13)

Check block (three minutes after kubectl apply -f speicherfresser.yaml):

amtool alert query alertname=PodBackOff

Expected:

Alertname   Starts At                Summary                           State
PodBackOff  2026-11-05 13:31:20 UTC  Pod in app startet immer wieder  active
kubectl -n app delete pod speicherfresser

Solution

The rule goes into the ConfigMap from Day 8 as a second file – or into a ConfigMap of its own with the label loki_rule:

cat > events-rules.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: events-log-rules
  namespace: monitoring
  labels:
    loki_rule: "1"
data:
  events.yaml: |
    groups:
      - name: kubernetes.events
        rules:
          - alert: PodBackOff
            expr: |
              sum by (namespace) (count_over_time(
                {job="kubernetes-events", namespace="app"}
                  |= "reason=BackOff" [5m])) > 0
            labels:
              severity: warning
            annotations:
              summary: Pod in {{ $labels.namespace }} startet immer wieder
EOF
kubectl apply -f events-rules.yaml
kubectl apply -f speicherfresser.yaml

The sidecar puts every file from every ConfigMap into /rules/fake. Two ConfigMaps, two files, one ruler.

Save your progress? Sign up for free

Exercise 4: Environment on Every Trace

Every trace that goes through the Collector should carry the resource attribute deployment.environment.name with the value lab – without changing notes-api. (Day 9, Day 10)

Check block (one minute after the helm upgrade):

curl -s -G localhost:30320/api/search --data-urlencode limit=1 \
  --data-urlencode 'q={ resource.deployment.environment.name = "lab" }' \
  | jq -r '.traces[0].rootServiceName'

Expected:

notes-api

Solution

A resource processor with action: upsert in the traces pipeline. Here is the whole otel.yaml:

cat > otel.yaml <<'EOF'
mode: deployment
fullnameOverride: otel-collector

image:
  repository: otel/opentelemetry-collector-k8s

presets:
  kubernetesAttributes:
    enabled: true

resources:
  requests: {cpu: 20m, memory: 64Mi}
  limits: {memory: 256Mi}

ports:
  jaeger-compact: {enabled: false}
  jaeger-thrift: {enabled: false}
  jaeger-grpc: {enabled: false}
  zipkin: {enabled: false}
  metrics: {enabled: true}

serviceMonitor:
  enabled: true

config:
  receivers:
    jaeger: null
    zipkin: null
    prometheus: null
  processors:
    resource/env:
      attributes:
        - key: deployment.environment.name
          value: lab
          action: upsert
  exporters:
    debug:
      verbosity: basic
    otlp_grpc/tempo:
      endpoint: tempo:4317
      tls:
        insecure: true
    otlp_http/loki:
      endpoint: http://loki:3100/otlp
    otlp_http/prometheus:
      endpoint: http://kps-prometheus:9090/api/v1/otlp
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, resource/env, batch]
        exporters: [debug, otlp_grpc/tempo]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_http/loki]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_http/prometheus]
EOF
helm upgrade otel open-telemetry/opentelemetry-collector --version 0.178.2 \
  -n monitoring -f otel.yaml --wait
  • resource changes resource attributes, attributes (a different processor) the attributes of individual spans.
  • The order matters: batch always comes last, so that what’s finished gets bundled.

Save your progress? Sign up for free

You made it

All four check blocks green? Then you can build, extend and operate an observability stack on Kubernetes – and, more importantly, use it when things are on fire.

Cleaning up is easy: multipass delete --purge obs on the laptop.