Skip to content
Hands-on IT IT Drills

Day 12: Observability

Goal: You read the stack’s logs, events, resources and healthchecks deliberately, and Prometheus collects notes-api’s metrics and raises an alert when the database is missing. · Duration: approx. 75 min · Prerequisite: Day 11

Briefing

  • Compose ships the basic tools: logs, events, stats, top, ps. They work on all services of the project or on individual ones.
  • notes-api writes metrics in the Prometheus text format under /metrics. Prometheus fetches them regularly (“scrape”) and stores them as time series.
  • A dedicated network monitoring connects only Prometheus, Grafana and api – Prometheus doesn’t need to see web or db.
  • Prometheus and Grafana are in the profile observability (Day 7): they only start when you explicitly ask for them.
                   network monitoring
 :9090 ─► prometheus ──scrape /metrics every 5 s──► api:8080 ──► db (backend)
 :3001 ─► grafana ────► prometheus:9090              ▲
                                                     └── :8080, proxy (frontend)

Drill 1: Reading logs deliberately

Start of the day: tear down the Day 11 stack, fetch the Day 11 state and start it:

source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag12 && cd ~/compose-lab/tag12
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag11/notes/." .
docker compose up -d --build --wait >/dev/null 2>&1
docker compose logs --tail 2 api worker
docker compose logs -t --since 30s worker

Expected (shortened, IDs and times differ on your machine):

worker-1  | worker started, api=http://api:8080, interval=10s
worker-1  | summary: 0 notes, 0 words
api-1     | time=2026-10-29T22:33:49.510Z level=INFO msg="store ready" store=postgres
api-1     | time=2026-10-29T22:33:49.511Z level=INFO msg=listening addr=:8080 …
worker-1  | 2026-10-29T22:33:52.727950424Z worker started, api=http://api:8080, …
worker-1  | 2026-10-29T22:33:52.947854585Z summary: 0 notes, 0 words
Switch Effect
--tail N only the last N lines per service
--since 30s / --until time window, also absolute (2026-10-29T22:00:00)
-t prepend Docker’s timestamp (for logs without their own time)
-f keep reading until Ctrl+C
--no-log-prefix without dienst-1 | (service-1 |) – the line stays machine-readable

Caddy writes JSON logs. Without the prefix they go straight into jq:

docker compose logs --no-log-prefix --tail 3 proxy | jq -c '{level, logger, msg}'

Expected (shortened):

{"level":"info","logger":"tls.obtain","msg":"obtaining certificate"}
{"level":"info","logger":"tls.obtain","msg":"certificate obtained successfully"}
{"level":"info","logger":"tls.obtain","msg":"releasing lock"}

logs works because the logging driver local (Day 9) stores and rotates the output locally. With a driver that only forwards (such as syslog), docker compose logs would stay empty.

Save your progress? Sign up for free

Drill 2: Recording events

docker compose events shows what Docker does with the project’s containers. Start the recording in the background and restart worker:

docker compose events --json > events.json &
sleep 1
docker compose restart worker
sleep 2
kill %1
jq -r '"\(.service)\t\(.action)"' events.json
head -1 events.json | jq '{service, action, time, signal: .attributes.signal}'
rm events.json

Expected (shortened, times differ):

 Container notes-worker-1 Restarting
 Container notes-worker-1 Started
worker  kill
worker  stop
worker  die
worker  start
worker  restart
api exec_create: /notes-api healthcheck
api exec_start: /notes-api healthcheck
api exec_die
{
  "service": "worker",
  "action": "kill",
  "time": "2026-10-29T23:34:09.808461475+01:00",
  "signal": "15"
}
  • restart is kill with signal 15 (SIGTERM), stop, die, start.
  • The exec_* lines are api’s healthcheck: every 10 seconds Docker runs /notes-api healthcheck in the container. That’s how you see whether and how often it runs.

Save your progress? Sign up for free

Drill 3: Resources with stats and top

docker compose stats --no-stream \
  --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}'
docker compose top api worker

Expected (shortened, values differ):

NAME             CPU %     MEM USAGE / LIMIT    PIDS
notes-proxy-1    0.00%     13.5MiB / 7.737GiB   9
notes-web-1      0.00%     20.02MiB / 128MiB    11
notes-worker-1   0.10%     23.18MiB / 128MiB    2
notes-api-1      0.00%     2.188MiB / 64MiB     7
notes-db-1       0.05%     32.11MiB / 512MiB    10
SERVICE  #   UID    PID     PPID    C   STIME  TTY  TIME      CMD
api      1   65532  136589  136565  0   23:33  ?    00:00:00  /notes-api
worker   1   65532  138407  138384  0   23:34  ?    00:00:00  /sbin/docker-init -- …
worker   1   65532  138437  138407  6   23:34  ?    00:00:00  /app/.venv/bin/python …
  • The limits from Day 9 show up as LIMIT. proxy has none and sees the entire memory of the Colima VM – a candidate for deploy.resources.limits.
  • top shows the processes with host PIDs and UIDs: api and worker run as 65532, worker under docker-init (init: true, Day 9).

Save your progress? Sign up for free

Drill 4: Evaluating healthchecks

ps as JSON returns State and Health per service; docker container inspect shows the history of the last checks:

docker compose ps --format json | jq -r '"\(.Service)\t\(.State)\t\(.Health)"'
docker container inspect --format '{{json .State.Health}}' notes-api-1 \
  | jq '{Status, FailingStreak, Last: (.Log[-1] | {ExitCode, Output})}'

Expected (shortened):

api running healthy
db  running healthy
proxy   running
web running healthy
worker  running
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Last": {
    "ExitCode": 0,
    "Output": ""
  }
}

proxy and worker have no healthcheck; Health stays empty. FailingStreak counts consecutive failures; when it reaches retries (default 3), the container becomes unhealthy. You’ll see this live in Drill 7.

Save your progress? Sign up for free

Drill 5: Adding Prometheus

Three files under observability/. First the scrape configuration:

mkdir -p observability
cat > observability/prometheus.yml <<'EOF'
# prometheus.yml – Day 12: scrape notes-api every 5 s
global:
  scrape_interval: 5s
  evaluation_interval: 5s

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  - job_name: notes-api
    static_configs:
      - targets: ["api:8080"]
EOF

Nobody has to specify the path /metrics; it’s the default. Then the alert rules:

cat > observability/alerts.yml <<'EOF'
# alerts.yml – Day 12: alert rules for notes
groups:
  - name: notes
    rules:
      - alert: NotesDBDown
        expr: notes_db_up == 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "notes-api erreicht die Datenbank nicht"
      - alert: NotesAPIDown
        expr: up{job="notes-api"} == 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Prometheus erreicht notes-api nicht"
EOF

Prometheus itself sets up for every target: 1 if the last scrape worked. notes_db_up comes from notes-api. The third file (Grafana) follows in Drill 8. In compose.yaml, add the network monitoring, attach api to it as well, and add the new service (complete in code/tag12/notes/compose.yaml):

  api:
    # … as on Day 11 …
    networks: [frontend, backend, monitoring]

  prometheus:
    <<: *hardening
    image: prom/prometheus:v3.15.1
    profiles: [observability]
    configs:
      - source: prometheus_yml
        target: /etc/prometheus/prometheus.yml
      - source: prometheus_alerts
        target: /etc/prometheus/alerts.yml
    volumes:
      - prometheus_data:/prometheus
    ports:
      - "127.0.0.1:${PROM_PORT:-9090}:9090"
    networks: [monitoring]
    restart: unless-stopped

Top level: under networks: the line monitoring: {}, under volumes: prometheus_data: {}, under configs: the two entries prometheus_yml and prometheus_alerts with file: ./observability/…. The network is not internal because Prometheus publishes a port (Day 2). promtool from the image checks both files:

docker pull -q prom/prometheus:v3.15.1
docker compose run --rm --no-deps --entrypoint promtool prometheus \
  check config /etc/prometheus/prometheus.yml 2>/dev/null

Expected:

Checking /etc/prometheus/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/alerts.yml
  SUCCESS: 2 rules found

run starts services with a profile even without --profile (Day 7). Now start only Prometheus:

docker compose up -d --wait prometheus >/dev/null 2>&1
sleep 3
curl -s localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | .health, .lastError'

Expected:

down
Get "http://api:8080/metrics": dial tcp: lookup api on 127.0.0.11:53: no such host

up -d prometheus only touched prometheus; api isn’t attached to the new network yet, and the name api is unknown there. Only an up across everything brings api in line:

docker compose --profile observability up -d --wait 2>&1 | grep 'api-1 Recreated'
sleep 6
curl -s localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | .health, .lastError'

Expected:

 Container notes-api-1 Recreated
up

Warning: --profile observability also starts Grafana (Drill 8, an image of about 1.2 GB). Without Grafana: docker compose up -d --wait and then docker compose up -d --wait prometheus.

Save your progress? Sign up for free

Drill 6: PromQL via the HTTP API

Prometheus has a web UI at http://localhost:9090. In the terminal you query the HTTP API; a small function prints labels and value:

q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
  | jq -r '.data.result[] | "\(.metric | del(.__name__, .instance, .job)
           | tostring)  \(.value[1])"'
}
q notes_build_info
q notes_db_up
for i in 1 2 3; do
  curl -s -o /dev/null -w '%{http_code} ' -X POST localhost:8080/notes \
    -H "X-API-Key: $(cat secrets/api_key.txt)" -H 'Content-Type: application/json' \
    -d "{\"text\":\"Metrik $i\"}"
done
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes -d '{}'
sleep 6
q notes_created_total
q 'sum by (path, code) (notes_http_requests_total)'
q 'sum by (path) (rate(notes_http_requests_total[1m]))'

Expected (shortened, numbers differ):

{"store":"postgres","version":"dev"}  1
{}  1
201 201 201 401
{}  3
{"code":"200","path":"/healthz"}  2
{"code":"200","path":"/notes"}  2
{"code":"200","path":"/metrics"}  3
{"code":"201","path":"/notes"}  3
{"code":"401","path":"/notes"}  1
{"path":"/healthz"}  0.024955306171504116
{"path":"/notes"}  0.024955306171504116
{"path":"/metrics"}  0.05814300151591712
  • notes_build_info is an “info metric”: the value is always 1, the information lives in the labels.
  • notes_http_requests_total is a counter; it only goes up. It becomes meaningful with rate(…[1m]): requests per second, averaged over one minute.
  • The /healthz calls are the Docker healthchecks; /notes with 200 are worker’s queries.

Save your progress? Sign up for free

Drill 7: Alert when the database is missing

curl -s localhost:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[] | "\(.name)\t\(.health)\t\(.state)"'
docker compose stop db
sleep 50
curl -s localhost:9090/api/v1/alerts \
  | jq -r '.data.alerts[] | "\(.labels.alertname)  \(.state)  \(.activeAt)"'
docker compose ps api --format '{{.Service}} {{.Status}}'
docker container inspect --format '{{json .State.Health}}' notes-api-1 \
  | jq '{Status, FailingStreak, Last: (.Log[-1] | {ExitCode, Output})}'

Expected (shortened, times differ):

NotesDBDown ok  inactive
NotesAPIDown    ok  inactive
 Container notes-db-1 Stopped
NotesDBDown  firing  2026-10-29T22:43:27.097381219Z
api Up 59 seconds (unhealthy)
{
  "Status": "unhealthy",
  "FailingStreak": 5,
  "Last": {
    "ExitCode": 1,
    "Output": ""
  }
}

The alert fires, and api is unhealthy (three failures in a row), but keeps running – Docker doesn’t restart unhealthy containers (Day 5). Prometheus can therefore keep scraping and sees notes_db_up 0. Bring the database back:

docker compose start db
sleep 20
curl -s localhost:9090/api/v1/alerts | jq -c '.data.alerts'
docker compose ps api db --format '{{.Service}} {{.Status}}'

Expected (shortened):

 Container notes-db-1 Started
[]
api Up About a minute (healthy)
db Up 20 seconds (healthy)

If you query notes_db_up and the alerts every five seconds during the 50 seconds, you see the transition (test run, loop not printed here):

  5s  db_up=1  alerts=
 10s  db_up=0  alerts=
 15s  db_up=0  alerts=NotesDBDown:pending
 …
 40s  db_up=0  alerts=NotesDBDown:pending
 45s  db_up=0  alerts=NotesDBDown:firing
  • notes_db_up only changes with the next healthcheck that checks the database (every 10 seconds).
  • for: 30s keeps the alert pending at first; only once the condition has held for 30 seconds does it become firing. That way, short blips don’t trigger an alert.
  • Prometheus only evaluates. Who gets notified (mail, chat, pager) is handled by an Alertmanager – see the Enterprise Note.

Save your progress? Sign up for free

Drill 8: Grafana (optional)

Grafana is the largest image in the book. Check the space beforehand:

df -h ~ | tail -1
docker pull -q grafana/grafana:13.3.0-slim
docker image ls grafana/grafana:13.3.0-slim --format '{{.Size}}'

Expected (shortened, values differ):

/dev/disk3s5   228Gi   152Gi    19Gi    90%    1.9M  197M    1%   /System/Volumes/Data
docker.io/grafana/grafana:13.3.0-slim
1.15GB

The admin password as a secret and the data source as a provisioned file:

printf '%s' 'lab-grafana-pw-123' > secrets/grafana_admin.txt
cat > observability/grafana-datasource.yaml <<'EOF'
# grafana-datasource.yaml – Day 12: Prometheus as the default data source
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
EOF
  grafana:
    <<: *hardening
    image: grafana/grafana:13.3.0-slim
    profiles: [observability]
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
      GF_PLUGINS_PREINSTALL_SYNC: prometheus
    secrets: [grafana_admin]
    configs:
      - source: grafana_datasource
        target: /etc/grafana/provisioning/datasources/prometheus.yaml
    volumes:
      - grafana_data:/var/lib/grafana
    tmpfs: [/tmp]
    ports:
      - "127.0.0.1:${GRAFANA_PORT:-3001}:3000"
    networks: [monitoring]
    depends_on: [prometheus]
    restart: unless-stopped

Add at the top level: grafana_data: {} under volumes:, grafana_datasource with file: ./observability/grafana-datasource.yaml under configs:, grafana_admin with file: ./secrets/grafana_admin.txt under secrets:.

  • Grafana reads any setting GF_<SECTION>_<KEY>__FILE from a file – so the password doesn’t appear in inspect.
  • The slim image does not contain the Prometheus plugin. At startup, Grafana downloads a list of recommended plugins from the Grafana catalog, in the background and about 570 MB in the volume (seen live). GF_PLUGINS_PREINSTALL_SYNC: prometheus loads the one plugin you need before the web server starts; without the line, the data source reports “Plugin not registered” for the first minute. For that, the network monitoring must be allowed to reach the internet.
  • The hardening from Day 9 applies in full: read_only with tmpfs: [/tmp], cap_drop: [ALL], no-new-privileges, user 472 from the image.
docker compose --profile observability up -d --wait >/dev/null 2>&1
sleep 5
curl -s localhost:3001/api/health | jq -c '{database, version}'
P="admin:$(cat secrets/grafana_admin.txt)"
curl -s -u "$P" localhost:3001/api/datasources | jq -c '.[] | {name, url, isDefault}'
uid=$(curl -s -u "$P" localhost:3001/api/datasources | jq -r '.[0].uid')
curl -s -u "$P" "localhost:3001/api/datasources/uid/$uid/health" | jq -r .message
curl -s -u "$P" \
  "localhost:3001/api/datasources/proxy/uid/$uid/api/v1/query?query=notes_db_up" \
  | jq -r '.data.result[0].value[1]'

Expected (shortened):

{"database":"ok","version":"13.3.0"}
{"name":"Prometheus","url":"http://prometheus:9090","isDefault":true}
Successfully queried the Prometheus API.
1

The last line is the same PromQL query as in Drill 6, this time through Grafana. In the browser: http://localhost:3001, log in with admin / lab-grafana-pw-123, and under “Explore” enter rate(notes_http_requests_total[1m]).

Save your progress? Sign up for free

Checkpoint

The stack is running with docker compose --profile observability up -d --wait (Grafana may be missing). You’ll find the complete state in code/tag12/notes/. The Prometheus configuration is valid:

cd ~/compose-lab/tag12
docker compose run --rm --no-deps --entrypoint promtool prometheus \
  check config /etc/prometheus/prometheus.yml

Expected (shortened):

Checking /etc/prometheus/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/alerts.yml
  SUCCESS: 2 rules found

Prometheus is ready:

curl -s localhost:9090/-/ready

Expected:

Prometheus Server is Ready.

The target notes-api is up, and so is the database:

curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="notes-api"}' \
  | jq -r '.data.result[0].value[1]'
curl -s localhost:9090/api/v1/query --data-urlencode 'query=notes_db_up' \
  | jq -r '.data.result[0].value[1]'

Expected:

1
1

Two alert rules loaded, none active:

curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts | length'

Expected:

NotesDBDown
NotesAPIDown
0

api is healthy and attached to the network monitoring:

docker compose ps api --format '{{.Status}} {{.Networks}}'

Expected (shortened):

Up 31 seconds (healthy) notes_backend,notes_frontend,notes_monitoring

Challenge

Trigger the alert NotesAPIDown without stopping a container or changing a file. Read in Prometheus why the scrape fails, and restore the desired state with a single Compose command.

Hint: docker network disconnect disconnects a running container from a network; on up, Compose compares a container’s networks with the file.

Solution

docker network disconnect notes_monitoring notes-api-1
sleep 50
curl -s localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | .health, .lastError'
curl -s localhost:9090/api/v1/alerts \
  | jq -r '.data.alerts[] | "\(.labels.alertname):\(.state)"'
docker compose ps api --format '{{.Service}} {{.Status}}'
docker compose up -d --wait 2>&1 | grep 'api-1 Recreated'
sleep 6
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[].health'

Expected (shortened):

down
Get "http://api:8080/metrics": dial tcp: lookup api on 127.0.0.11:53: no such host
NotesAPIDown:firing
api Up 5 minutes (healthy)
 Container notes-api-1 Recreated
up

As far as Docker is concerned, api is perfectly healthy – the healthcheck runs inside the container and notices nothing of a missing network. Only the view from the outside (Prometheus) shows the problem. That’s why healthchecks and monitoring complement each other rather than replace each other. docker compose up -d detects the missing network and recreates api (verified live in Compose 5.6.0). Finally:

docker compose --profile observability down

Without --profile, down would leave Prometheus and Grafana running, and the network notes_monitoring would be left over with “Resource is still in use” (verified live).

Save your progress? Sign up for free

Enterprise Note

  • Logs should be collected centrally (Loki, Elasticsearch/OpenSearch, Splunk): via a logging driver, or better via an agent (Fluent Bit, Vector, Grafana Alloy) that reads the local files – then docker compose logs keeps working.
  • For notifications, Prometheus needs an Alertmanager (routing, grouping, silencing), and for long-term retention remote write (Mimir, Thanos, VictoriaMetrics).
  • Never expose /metrics publicly (Day 11 blocks it at the proxy). Run Prometheus and Grafana only with login, TLS and dedicated networks; replace default passwords immediately.
  • Grafana in isolated environments: don’t load plugins from the internet, but bake them into the image beforehand or pull them from an internal catalog; GF_PLUGINS_PREINSTALL_DISABLED turns off the downloading (live: then also PREINSTALL_SYNC).
  • For traces and standardized metrics, OpenTelemetry is becoming the standard: applications send to a collector, which distributes to Prometheus, Tempo/Jaeger and Loki.