Day 8: Structured Logs and Log Alerts
Goal: notes-api writes JSON, you parse and format it with LogQL, measure the cardinality of your streams and let Loki trigger alerts itself. · Duration: approx. 60 min · Prerequisite: Day 7
Briefing
- Structured logs are logs with fixed fields instead of free text. JSON is the most widespread format: every tool can read it, no field gets lost when parsing.
- In Loki, every combination of labels is its own stream. Few streams with many lines are cheap, many streams with few lines are expensive. The number of streams is called cardinality.
- Loki has a ruler: it evaluates LogQL rules regularly and sends alerts to the same Alertmanager as Prometheus.
- That way you capture what no metric counts – such as a warning that appears in the log only once at startup.
notes-api ── JSON ──► Alloy ──► Loki ── ruler (rules from ConfigMaps)
│
▼
Alertmanager ──► alarm-echo
Drill 1: JSON Instead of Text
logging.go knows both formats. The environment variable
decides:
kubectl -n app set env deploy/notes-api NOTES_LOG_FORMAT=json
kubectl -n app rollout status deploy/notes-api
kubectl -n app logs deploy/notes-api --tail 2Expected (shortened):
deployment "notes-api" successfully rolled out
{"time":"2026-11-05T09:17:08.338477173Z","level":"INFO","msg":"request","method":"GET...
{"time":"2026-11-05T09:17:08.840240332Z","level":"INFO","msg":"request","method":"GET...
- The same fields as before, only as JSON.
statusandduration_msare now real numbers, not strings. - Nothing changed in the code:
slogseparates what is logged from the format.slog.NewJSONHandlerinstead ofslog.NewTextHandler– that’s all there is to it.
Save your progress? Sign up for free
Drill 2: Parsing and Formatting
logcli query --quiet --limit 3 -o raw --since 1m \
'{app="notes-api"} | json | method="POST"
| line_format "{{.method}} {{.path}} {{.status}} {{.duration_ms}}ms"'Expected (the times differ):
POST /notes 201 14ms
POST /notes 201 3ms
POST /notes 201 1ms
| jsonturns every field into a label – like| logfmton Day 7.method="POST"filters by the label from the parser. It comes after| jsonbecause the label doesn’t exist before that.line_formatrebuilds the line, with Go templates as in the alert annotations. For a report or a Grafana panel, only what matters remains.
Fields can also be turned into numbers. The average duration of
GET /notes in milliseconds:
logcli instant-query --quiet \
'avg(avg_over_time({app="notes-api"} | json | method="GET"
| unwrap duration_ms [5m]))'Expected (roughly):
[
{
"metric": {},
"value": [
1793870229.83,
"8.65991902834008"
]
}
]
unwrap duration_mstakes the value of the field instead of the number of lines.avg_over_timeaverages it over five minutes.- After a few minutes, Prometheus delivers the same number too. The histogram from Day 1 is much cheaper for that – evaluating logs means reading every line.
Save your progress? Sign up for free
Drill 3: Cardinality
How many streams does Loki have for the namespace
app?
logcli series --quiet --analyze-labels '{namespace="app"}' --since 1hExpected (the numbers differ):
Total Streams: 27
Unique Labels: 7
Label Name Unique Values Found In Streams
instance 27 27
pod 14 27
container 5 27
service_name 3 27
app 3 27
job 1 27
namespace 1 27
- 27 streams for three applications in one hour. The driver is
pod: every rollout creates new pod names and thus new streams. The old ones stop growing but stay in the index until retention. instancehas even more values because the container is part of it.- Imagine Alloy set
statusorpathas a label: every new value multiplies the streams. With a user ID, there would be millions.
That’s why the rule in Loki is: labels describe where a line comes from. What it says stays in the text and is parsed at query time.
Save your progress? Sign up for free
Drill 4: The Ruler
The ruler already runs inside Loki but knows neither rules nor
Alertmanager. Extend loki.yaml – here is the whole
file:
cd ~/obs-lab
cat > loki.yaml <<'EOF'
deploymentMode: Monolithic
loki:
auth_enabled: false
commonConfig:
replication_factor: 1
storage:
type: filesystem
schemaConfig:
configs:
- from: "2026-01-01"
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
retention_period: 72h
compactor:
retention_enabled: true
delete_request_store: filesystem
rulerConfig:
storage:
type: local
local:
directory: /rules
rule_path: /var/loki/rules-temp
alertmanager_url: http://kps-alertmanager.monitoring:9093
enable_api: true
wal:
dir: /var/loki/ruler-wal
singleBinary:
replicas: 1
persistence:
size: 2Gi
resources:
requests: {cpu: 100m, memory: 128Mi}
limits: {memory: 512Mi}
read: {replicas: 0}
write: {replicas: 0}
backend: {replicas: 0}
gateway: {enabled: false}
chunksCache: {enabled: false}
resultsCache: {enabled: false}
lokiCanary: {enabled: false}
test: {enabled: false}
sidecar:
rules:
folder: /rules/fake
EOF
helm upgrade loki oci://ghcr.io/grafana-community/helm-charts/loki --version 20.4.1 \
-n monitoring -f loki.yaml --waitExpected (shortened):
Pulled: ghcr.io/grafana-community/helm-charts/loki:20.4.1
...
STATUS: deployed
REVISION: 2
rulerConfig.storagetells the ruler that its rules are files under/rules. That’s where the sidecar from Day 7 writes to.- Without multi-tenancy, the only tenant in Loki is called
fake. Rules live in a subfolder per tenant – hencefolder: /rules/fakefor the sidecar. alertmanager_urlpoints to the same Alertmanager as Prometheus.enable_apiallows querying the loaded rules.
Save your progress? Sign up for free
Drill 5: An Alert from the Log
faults.go writes faults enabled once at
startup when a fault is configured. There is no metric for that – but
there is a log line:
cat > notes-log-rules.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: notes-api-log-rules
namespace: monitoring
labels:
loki_rule: "1"
data:
notes-api.yaml: |
groups:
- name: notes-api.logs
rules:
- alert: NotesApiFaultsEnabled
expr: |
sum by (namespace) (count_over_time(
{app="notes-api"} | json | msg="faults enabled" [10m])) > 0
labels:
severity: info
annotations:
summary: notes-api läuft mit eingebauten Störungen
EOF
kubectl apply -f notes-log-rules.yamlExpected:
configmap/notes-api-log-rules created
- The format is the same as in Prometheus: groups, rules,
expr, labels, annotations. Only the query is LogQL. - The line appears once per pod in the log.
[10m]keeps the alert active for ten minutes after the last start. After that it goes out, even if the fault is still configured.
After a minute, the ruler knows the rule:
curl -s localhost:30310/prometheus/api/v1/rules \
| jq -c '.data.groups[].rules[] | [.name, .state, .health]'Expected:
["NotesApiFaultsEnabled","inactive","ok"]
Under /prometheus/api/v1/rules, Loki answers in the same
format as Prometheus. That’s why Grafana shows rules from both in the
same place.
Now a small fault:
kubectl -n app set env deploy/notes-api NOTES_DELAY=50ms
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
After about a minute:
amtool alert query alertname=NotesApiFaultsEnabledExpected:
Alertname Starts At Summary...
NotesApiFaultsEnabled 2026-11-05 09:19:23 UTC notes-api läuft mit eingebauten Störu...
It wasn’t Prometheus that triggered the alert but Loki. For Alertmanager that makes no difference – alarm-echo gets it too.
Take back the fault:
kubectl -n app set env deploy/notes-api NOTES_DELAY-Save your progress? Sign up for free
Checkpoint
kubectl -n app set env deploy/notes-api --list | grep LOG_FORMATExpected:
NOTES_LOG_FORMAT=json
NOTES_LOG_FORMAT=json
Once for notes-api, once for the init container migrate
– set env changes all containers. JSON stays on from now
on.
kubectl -n monitoring logs deploy/alarm-echo | grep '^{' \
| jq -r '.json.alerts[].labels.alertname' | sort -uExpected:
NotesApiDown
NotesApiErrors
NotesApiFaultsEnabled
NotesApiSlow
All alerts in this book so far, three from Prometheus, one from Loki.
Challenge
Write a second log rule NotesApiServerErrors: it
should fire when notes-api writes more than 20 lines with
level ERROR in five minutes. Trigger it with
NOTES_ERROR_RATE=0.2.
Hint: A ConfigMap may contain several rules. Change the file and apply it again.
Solution
cat > notes-log-rules.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: notes-api-log-rules
namespace: monitoring
labels:
loki_rule: "1"
data:
notes-api.yaml: |
groups:
- name: notes-api.logs
rules:
- alert: NotesApiFaultsEnabled
expr: |
sum by (namespace) (count_over_time(
{app="notes-api"} | json | msg="faults enabled" [10m])) > 0
labels:
severity: info
annotations:
summary: notes-api läuft mit eingebauten Störungen
- alert: NotesApiServerErrors
expr: |
sum by (namespace) (count_over_time(
{app="notes-api"} | json | level="ERROR" [5m])) > 20
labels:
severity: warning
annotations:
summary: "notes-api: {{ $value }} Fehlerzeilen in 5 Minuten"
EOF
kubectl apply -f notes-log-rules.yaml
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2After two minutes:
amtool alert query alertname=NotesApiServerErrorsExpected:
Alertname Starts At Summary...
NotesApiServerErrors 2026-11-05 09:22:23 UTC notes-api: 178 Fehlerzeilen in 5 Minut...
Take it back:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-Metric or log? NotesApiErrors from Day 4 measures the same thing more precisely and more cheaply. Log rules pay off for events that no metric knows about.
Save your progress? Sign up for free
Enterprise Note
- Define the log format: A common JSON schema for all applications – field names for time, level, message, trace ID – saves a special case in every query. OpenTelemetry defines the Semantic Conventions for that.
- Structured metadata: Loki 3 can store fields like
trace_idnext to the line without turning them into labels. They are filterable but don’t increase cardinality. - Keep an eye on costs:
logcli series --analyze-labelsbelongs in every regular review. A single wrong label can multiply the storage of a Loki.