Day 12: Observability
Goal: You read the stack’s logs, events, resources and healthchecks deliberately, and Prometheus collects notes-api’s metrics and raises an alert when the database is missing. · Duration: approx. 75 min · Prerequisite: Day 11
Briefing
- Compose ships the basic tools:
logs,events,stats,top,ps. They work on all services of the project or on individual ones. - notes-api writes metrics in the Prometheus text
format under
/metrics. Prometheus fetches them regularly (“scrape”) and stores them as time series. - A dedicated network
monitoringconnects only Prometheus, Grafana and api – Prometheus doesn’t need to see web or db. - Prometheus and Grafana are in the profile
observability(Day 7): they only start when you explicitly ask for them.
network monitoring
:9090 ─► prometheus ──scrape /metrics every 5 s──► api:8080 ──► db (backend)
:3001 ─► grafana ────► prometheus:9090 ▲
└── :8080, proxy (frontend)
Drill 1: Reading logs deliberately
Start of the day: tear down the Day 11 stack, fetch the Day 11 state and start it:
source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag12 && cd ~/compose-lab/tag12
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag11/notes/." .
docker compose up -d --build --wait >/dev/null 2>&1
docker compose logs --tail 2 api worker
docker compose logs -t --since 30s workerExpected (shortened, IDs and times differ on your machine):
worker-1 | worker started, api=http://api:8080, interval=10s
worker-1 | summary: 0 notes, 0 words
api-1 | time=2026-10-29T22:33:49.510Z level=INFO msg="store ready" store=postgres
api-1 | time=2026-10-29T22:33:49.511Z level=INFO msg=listening addr=:8080 …
worker-1 | 2026-10-29T22:33:52.727950424Z worker started, api=http://api:8080, …
worker-1 | 2026-10-29T22:33:52.947854585Z summary: 0 notes, 0 words
| Switch | Effect |
|---|---|
--tail N |
only the last N lines per service |
--since 30s / --until |
time window, also absolute (2026-10-29T22:00:00) |
-t |
prepend Docker’s timestamp (for logs without their own time) |
-f |
keep reading until Ctrl+C |
--no-log-prefix |
without dienst-1 | (service-1 |) – the
line stays machine-readable |
Caddy writes JSON logs. Without the prefix they go straight into
jq:
docker compose logs --no-log-prefix --tail 3 proxy | jq -c '{level, logger, msg}'Expected (shortened):
{"level":"info","logger":"tls.obtain","msg":"obtaining certificate"}
{"level":"info","logger":"tls.obtain","msg":"certificate obtained successfully"}
{"level":"info","logger":"tls.obtain","msg":"releasing lock"}
logs works because the logging driver local
(Day 9) stores and rotates the output locally. With a driver that only
forwards (such as syslog), docker compose logs
would stay empty.
Save your progress? Sign up for free
Drill 2: Recording events
docker compose events shows what Docker does with the
project’s containers. Start the recording in the background and restart
worker:
docker compose events --json > events.json &
sleep 1
docker compose restart worker
sleep 2
kill %1
jq -r '"\(.service)\t\(.action)"' events.json
head -1 events.json | jq '{service, action, time, signal: .attributes.signal}'
rm events.jsonExpected (shortened, times differ):
Container notes-worker-1 Restarting
Container notes-worker-1 Started
worker kill
worker stop
worker die
worker start
worker restart
api exec_create: /notes-api healthcheck
api exec_start: /notes-api healthcheck
api exec_die
{
"service": "worker",
"action": "kill",
"time": "2026-10-29T23:34:09.808461475+01:00",
"signal": "15"
}
restartiskillwith signal 15 (SIGTERM),stop,die,start.- The
exec_*lines are api’s healthcheck: every 10 seconds Docker runs/notes-api healthcheckin the container. That’s how you see whether and how often it runs.
Save your progress? Sign up for free
Drill 3: Resources with stats and top
docker compose stats --no-stream \
--format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}'
docker compose top api workerExpected (shortened, values differ):
NAME CPU % MEM USAGE / LIMIT PIDS
notes-proxy-1 0.00% 13.5MiB / 7.737GiB 9
notes-web-1 0.00% 20.02MiB / 128MiB 11
notes-worker-1 0.10% 23.18MiB / 128MiB 2
notes-api-1 0.00% 2.188MiB / 64MiB 7
notes-db-1 0.05% 32.11MiB / 512MiB 10
SERVICE # UID PID PPID C STIME TTY TIME CMD
api 1 65532 136589 136565 0 23:33 ? 00:00:00 /notes-api
worker 1 65532 138407 138384 0 23:34 ? 00:00:00 /sbin/docker-init -- …
worker 1 65532 138437 138407 6 23:34 ? 00:00:00 /app/.venv/bin/python …
- The limits from Day 9 show up as
LIMIT. proxy has none and sees the entire memory of the Colima VM – a candidate fordeploy.resources.limits. topshows the processes with host PIDs and UIDs: api and worker run as 65532, worker underdocker-init(init: true, Day 9).
Save your progress? Sign up for free
Drill 4: Evaluating healthchecks
ps as JSON returns State and
Health per service; docker container inspect
shows the history of the last checks:
docker compose ps --format json | jq -r '"\(.Service)\t\(.State)\t\(.Health)"'
docker container inspect --format '{{json .State.Health}}' notes-api-1 \
| jq '{Status, FailingStreak, Last: (.Log[-1] | {ExitCode, Output})}'Expected (shortened):
api running healthy
db running healthy
proxy running
web running healthy
worker running
{
"Status": "healthy",
"FailingStreak": 0,
"Last": {
"ExitCode": 0,
"Output": ""
}
}
proxy and worker have no healthcheck; Health stays
empty. FailingStreak counts consecutive failures; when it
reaches retries (default 3), the container becomes
unhealthy. You’ll see this live in Drill 7.
Save your progress? Sign up for free
Drill 5: Adding Prometheus
Three files under observability/. First the scrape
configuration:
mkdir -p observability
cat > observability/prometheus.yml <<'EOF'
# prometheus.yml – Day 12: scrape notes-api every 5 s
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/alerts.yml
scrape_configs:
- job_name: notes-api
static_configs:
- targets: ["api:8080"]
EOFNobody has to specify the path /metrics; it’s the
default. Then the alert rules:
cat > observability/alerts.yml <<'EOF'
# alerts.yml – Day 12: alert rules for notes
groups:
- name: notes
rules:
- alert: NotesDBDown
expr: notes_db_up == 0
for: 30s
labels:
severity: critical
annotations:
summary: "notes-api erreicht die Datenbank nicht"
- alert: NotesAPIDown
expr: up{job="notes-api"} == 0
for: 30s
labels:
severity: critical
annotations:
summary: "Prometheus erreicht notes-api nicht"
EOFPrometheus itself sets up for every target: 1 if the
last scrape worked. notes_db_up comes from notes-api. The
third file (Grafana) follows in Drill 8. In compose.yaml,
add the network monitoring, attach api to it as well, and
add the new service (complete in
code/tag12/notes/compose.yaml):
api:
# … as on Day 11 …
networks: [frontend, backend, monitoring]
prometheus:
<<: *hardening
image: prom/prometheus:v3.15.1
profiles: [observability]
configs:
- source: prometheus_yml
target: /etc/prometheus/prometheus.yml
- source: prometheus_alerts
target: /etc/prometheus/alerts.yml
volumes:
- prometheus_data:/prometheus
ports:
- "127.0.0.1:${PROM_PORT:-9090}:9090"
networks: [monitoring]
restart: unless-stoppedTop level: under networks: the line
monitoring: {}, under volumes:
prometheus_data: {}, under configs: the two
entries prometheus_yml and prometheus_alerts
with file: ./observability/…. The network is not
internal because Prometheus publishes a port (Day 2).
promtool from the image checks both files:
docker pull -q prom/prometheus:v3.15.1
docker compose run --rm --no-deps --entrypoint promtool prometheus \
check config /etc/prometheus/prometheus.yml 2>/dev/nullExpected:
Checking /etc/prometheus/prometheus.yml
SUCCESS: 1 rule files found
SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Checking /etc/prometheus/alerts.yml
SUCCESS: 2 rules found
run starts services with a profile even without
--profile (Day 7). Now start only Prometheus:
docker compose up -d --wait prometheus >/dev/null 2>&1
sleep 3
curl -s localhost:9090/api/v1/targets \
| jq -r '.data.activeTargets[] | .health, .lastError'Expected:
down
Get "http://api:8080/metrics": dial tcp: lookup api on 127.0.0.11:53: no such host
up -d prometheus only touched prometheus; api isn’t
attached to the new network yet, and the name api is
unknown there. Only an up across everything brings api in
line:
docker compose --profile observability up -d --wait 2>&1 | grep 'api-1 Recreated'
sleep 6
curl -s localhost:9090/api/v1/targets \
| jq -r '.data.activeTargets[] | .health, .lastError'Expected:
Container notes-api-1 Recreated
up
Warning:
--profile observabilityalso starts Grafana (Drill 8, an image of about 1.2 GB). Without Grafana:docker compose up -d --waitand thendocker compose up -d --wait prometheus.
Save your progress? Sign up for free
Drill 6: PromQL via the HTTP API
Prometheus has a web UI at http://localhost:9090. In the
terminal you query the HTTP API; a small function prints labels and
value:
q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
| jq -r '.data.result[] | "\(.metric | del(.__name__, .instance, .job)
| tostring) \(.value[1])"'
}
q notes_build_info
q notes_db_up
for i in 1 2 3; do
curl -s -o /dev/null -w '%{http_code} ' -X POST localhost:8080/notes \
-H "X-API-Key: $(cat secrets/api_key.txt)" -H 'Content-Type: application/json' \
-d "{\"text\":\"Metrik $i\"}"
done
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes -d '{}'
sleep 6
q notes_created_total
q 'sum by (path, code) (notes_http_requests_total)'
q 'sum by (path) (rate(notes_http_requests_total[1m]))'Expected (shortened, numbers differ):
{"store":"postgres","version":"dev"} 1
{} 1
201 201 201 401
{} 3
{"code":"200","path":"/healthz"} 2
{"code":"200","path":"/notes"} 2
{"code":"200","path":"/metrics"} 3
{"code":"201","path":"/notes"} 3
{"code":"401","path":"/notes"} 1
{"path":"/healthz"} 0.024955306171504116
{"path":"/notes"} 0.024955306171504116
{"path":"/metrics"} 0.05814300151591712
notes_build_infois an “info metric”: the value is always 1, the information lives in the labels.notes_http_requests_totalis a counter; it only goes up. It becomes meaningful withrate(…[1m]): requests per second, averaged over one minute.- The
/healthzcalls are the Docker healthchecks;/noteswith 200 are worker’s queries.
Save your progress? Sign up for free
Drill 7: Alert when the database is missing
curl -s localhost:9090/api/v1/rules \
| jq -r '.data.groups[].rules[] | "\(.name)\t\(.health)\t\(.state)"'
docker compose stop db
sleep 50
curl -s localhost:9090/api/v1/alerts \
| jq -r '.data.alerts[] | "\(.labels.alertname) \(.state) \(.activeAt)"'
docker compose ps api --format '{{.Service}} {{.Status}}'
docker container inspect --format '{{json .State.Health}}' notes-api-1 \
| jq '{Status, FailingStreak, Last: (.Log[-1] | {ExitCode, Output})}'Expected (shortened, times differ):
NotesDBDown ok inactive
NotesAPIDown ok inactive
Container notes-db-1 Stopped
NotesDBDown firing 2026-10-29T22:43:27.097381219Z
api Up 59 seconds (unhealthy)
{
"Status": "unhealthy",
"FailingStreak": 5,
"Last": {
"ExitCode": 1,
"Output": ""
}
}
The alert fires, and api is unhealthy (three failures in
a row), but keeps running – Docker doesn’t restart
unhealthy containers (Day 5). Prometheus can therefore keep
scraping and sees notes_db_up 0. Bring the database
back:
docker compose start db
sleep 20
curl -s localhost:9090/api/v1/alerts | jq -c '.data.alerts'
docker compose ps api db --format '{{.Service}} {{.Status}}'Expected (shortened):
Container notes-db-1 Started
[]
api Up About a minute (healthy)
db Up 20 seconds (healthy)
If you query notes_db_up and the alerts every five
seconds during the 50 seconds, you see the transition (test run, loop
not printed here):
5s db_up=1 alerts=
10s db_up=0 alerts=
15s db_up=0 alerts=NotesDBDown:pending
…
40s db_up=0 alerts=NotesDBDown:pending
45s db_up=0 alerts=NotesDBDown:firing
notes_db_uponly changes with the next healthcheck that checks the database (every 10 seconds).for: 30skeeps the alert pending at first; only once the condition has held for 30 seconds does it become firing. That way, short blips don’t trigger an alert.- Prometheus only evaluates. Who gets notified (mail, chat, pager) is handled by an Alertmanager – see the Enterprise Note.
Save your progress? Sign up for free
Drill 8: Grafana (optional)
Grafana is the largest image in the book. Check the space beforehand:
df -h ~ | tail -1
docker pull -q grafana/grafana:13.3.0-slim
docker image ls grafana/grafana:13.3.0-slim --format '{{.Size}}'Expected (shortened, values differ):
/dev/disk3s5 228Gi 152Gi 19Gi 90% 1.9M 197M 1% /System/Volumes/Data
docker.io/grafana/grafana:13.3.0-slim
1.15GB
The admin password as a secret and the data source as a provisioned file:
printf '%s' 'lab-grafana-pw-123' > secrets/grafana_admin.txt
cat > observability/grafana-datasource.yaml <<'EOF'
# grafana-datasource.yaml – Day 12: Prometheus as the default data source
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
EOF grafana:
<<: *hardening
image: grafana/grafana:13.3.0-slim
profiles: [observability]
environment:
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
GF_PLUGINS_PREINSTALL_SYNC: prometheus
secrets: [grafana_admin]
configs:
- source: grafana_datasource
target: /etc/grafana/provisioning/datasources/prometheus.yaml
volumes:
- grafana_data:/var/lib/grafana
tmpfs: [/tmp]
ports:
- "127.0.0.1:${GRAFANA_PORT:-3001}:3000"
networks: [monitoring]
depends_on: [prometheus]
restart: unless-stoppedAdd at the top level: grafana_data: {} under
volumes:, grafana_datasource with
file: ./observability/grafana-datasource.yaml under
configs:, grafana_admin with
file: ./secrets/grafana_admin.txt under
secrets:.
- Grafana reads any setting
GF_<SECTION>_<KEY>__FILEfrom a file – so the password doesn’t appear ininspect. - The slim image does not contain the Prometheus
plugin. At startup, Grafana downloads a list of recommended plugins from
the Grafana catalog, in the background and about 570 MB in the volume
(seen live).
GF_PLUGINS_PREINSTALL_SYNC: prometheusloads the one plugin you need before the web server starts; without the line, the data source reports “Plugin not registered” for the first minute. For that, the networkmonitoringmust be allowed to reach the internet. - The hardening from Day 9 applies in full:
read_onlywithtmpfs: [/tmp],cap_drop: [ALL],no-new-privileges, user 472 from the image.
docker compose --profile observability up -d --wait >/dev/null 2>&1
sleep 5
curl -s localhost:3001/api/health | jq -c '{database, version}'
P="admin:$(cat secrets/grafana_admin.txt)"
curl -s -u "$P" localhost:3001/api/datasources | jq -c '.[] | {name, url, isDefault}'
uid=$(curl -s -u "$P" localhost:3001/api/datasources | jq -r '.[0].uid')
curl -s -u "$P" "localhost:3001/api/datasources/uid/$uid/health" | jq -r .message
curl -s -u "$P" \
"localhost:3001/api/datasources/proxy/uid/$uid/api/v1/query?query=notes_db_up" \
| jq -r '.data.result[0].value[1]'Expected (shortened):
{"database":"ok","version":"13.3.0"}
{"name":"Prometheus","url":"http://prometheus:9090","isDefault":true}
Successfully queried the Prometheus API.
1
The last line is the same PromQL query as in Drill 6, this time
through Grafana. In the browser: http://localhost:3001, log
in with admin / lab-grafana-pw-123, and under
“Explore” enter rate(notes_http_requests_total[1m]).
Save your progress? Sign up for free
Checkpoint
The stack is running with
docker compose --profile observability up -d --wait
(Grafana may be missing). You’ll find the complete state in
code/tag12/notes/. The Prometheus configuration is
valid:
cd ~/compose-lab/tag12
docker compose run --rm --no-deps --entrypoint promtool prometheus \
check config /etc/prometheus/prometheus.ymlExpected (shortened):
Checking /etc/prometheus/prometheus.yml
SUCCESS: 1 rule files found
SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Checking /etc/prometheus/alerts.yml
SUCCESS: 2 rules found
Prometheus is ready:
curl -s localhost:9090/-/readyExpected:
Prometheus Server is Ready.
The target notes-api is up, and so is the database:
curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="notes-api"}' \
| jq -r '.data.result[0].value[1]'
curl -s localhost:9090/api/v1/query --data-urlencode 'query=notes_db_up' \
| jq -r '.data.result[0].value[1]'Expected:
1
1
Two alert rules loaded, none active:
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts | length'Expected:
NotesDBDown
NotesAPIDown
0
api is healthy and attached to the network
monitoring:
docker compose ps api --format '{{.Status}} {{.Networks}}'Expected (shortened):
Up 31 seconds (healthy) notes_backend,notes_frontend,notes_monitoring
Challenge
Trigger the alert NotesAPIDown without stopping a
container or changing a file. Read in Prometheus why the scrape fails,
and restore the desired state with a single Compose command.
Hint: docker network disconnect
disconnects a running container from a network; on up,
Compose compares a container’s networks with the file.
Solution
docker network disconnect notes_monitoring notes-api-1
sleep 50
curl -s localhost:9090/api/v1/targets \
| jq -r '.data.activeTargets[] | .health, .lastError'
curl -s localhost:9090/api/v1/alerts \
| jq -r '.data.alerts[] | "\(.labels.alertname):\(.state)"'
docker compose ps api --format '{{.Service}} {{.Status}}'
docker compose up -d --wait 2>&1 | grep 'api-1 Recreated'
sleep 6
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[].health'Expected (shortened):
down
Get "http://api:8080/metrics": dial tcp: lookup api on 127.0.0.11:53: no such host
NotesAPIDown:firing
api Up 5 minutes (healthy)
Container notes-api-1 Recreated
up
As far as Docker is concerned, api is perfectly healthy – the
healthcheck runs inside the container and notices nothing of a missing
network. Only the view from the outside (Prometheus) shows the problem.
That’s why healthchecks and monitoring complement each other rather than
replace each other. docker compose up -d detects the
missing network and recreates api (verified live in Compose 5.6.0).
Finally:
docker compose --profile observability downWithout --profile, down would leave
Prometheus and Grafana running, and the network
notes_monitoring would be left over with “Resource is still
in use” (verified live).
Save your progress? Sign up for free
Enterprise Note
- Logs should be collected centrally (Loki, Elasticsearch/OpenSearch,
Splunk): via a logging driver, or better via an agent (Fluent Bit,
Vector, Grafana Alloy) that reads the local files – then
docker compose logskeeps working. - For notifications, Prometheus needs an Alertmanager (routing, grouping, silencing), and for long-term retention remote write (Mimir, Thanos, VictoriaMetrics).
- Never expose
/metricspublicly (Day 11 blocks it at the proxy). Run Prometheus and Grafana only with login, TLS and dedicated networks; replace default passwords immediately. - Grafana in isolated environments: don’t load plugins from the
internet, but bake them into the image beforehand or pull them from an
internal catalog;
GF_PLUGINS_PREINSTALL_DISABLEDturns off the downloading (live: then alsoPREINSTALL_SYNC). - For traces and standardized metrics, OpenTelemetry is becoming the standard: applications send to a collector, which distributes to Prometheus, Tempo/Jaeger and Loki.