Skip to content
Hands-on IT IT Drills

Day 8: Structured Logs and Log Alerts

Goal: notes-api writes JSON, you parse and format it with LogQL, measure the cardinality of your streams and let Loki trigger alerts itself. · Duration: approx. 60 min · Prerequisite: Day 7

Briefing

  • Structured logs are logs with fixed fields instead of free text. JSON is the most widespread format: every tool can read it, no field gets lost when parsing.
  • In Loki, every combination of labels is its own stream. Few streams with many lines are cheap, many streams with few lines are expensive. The number of streams is called cardinality.
  • Loki has a ruler: it evaluates LogQL rules regularly and sends alerts to the same Alertmanager as Prometheus.
  • That way you capture what no metric counts – such as a warning that appears in the log only once at startup.
 notes-api ── JSON ──► Alloy ──► Loki ── ruler (rules from ConfigMaps)
                                             │
                                             ▼
                                        Alertmanager ──► alarm-echo

Drill 1: JSON Instead of Text

logging.go knows both formats. The environment variable decides:

kubectl -n app set env deploy/notes-api NOTES_LOG_FORMAT=json
kubectl -n app rollout status deploy/notes-api
kubectl -n app logs deploy/notes-api --tail 2

Expected (shortened):

deployment "notes-api" successfully rolled out
{"time":"2026-11-05T09:17:08.338477173Z","level":"INFO","msg":"request","method":"GET...
{"time":"2026-11-05T09:17:08.840240332Z","level":"INFO","msg":"request","method":"GET...
  • The same fields as before, only as JSON. status and duration_ms are now real numbers, not strings.
  • Nothing changed in the code: slog separates what is logged from the format. slog.NewJSONHandler instead of slog.NewTextHandler – that’s all there is to it.

Save your progress? Sign up for free

Drill 2: Parsing and Formatting

logcli query --quiet --limit 3 -o raw --since 1m \
  '{app="notes-api"} | json | method="POST"
     | line_format "{{.method}} {{.path}} {{.status}} {{.duration_ms}}ms"'

Expected (the times differ):

POST /notes 201 14ms
POST /notes 201 3ms
POST /notes 201 1ms
  • | json turns every field into a label – like | logfmt on Day 7.
  • method="POST" filters by the label from the parser. It comes after | json because the label doesn’t exist before that.
  • line_format rebuilds the line, with Go templates as in the alert annotations. For a report or a Grafana panel, only what matters remains.

Fields can also be turned into numbers. The average duration of GET /notes in milliseconds:

logcli instant-query --quiet \
  'avg(avg_over_time({app="notes-api"} | json | method="GET"
     | unwrap duration_ms [5m]))'

Expected (roughly):

[
  {
    "metric": {},
    "value": [
      1793870229.83,
      "8.65991902834008"
    ]
  }
]
  • unwrap duration_ms takes the value of the field instead of the number of lines. avg_over_time averages it over five minutes.
  • After a few minutes, Prometheus delivers the same number too. The histogram from Day 1 is much cheaper for that – evaluating logs means reading every line.

Save your progress? Sign up for free

Drill 3: Cardinality

How many streams does Loki have for the namespace app?

logcli series --quiet --analyze-labels '{namespace="app"}' --since 1h

Expected (the numbers differ):

Total Streams:  27
Unique Labels:  7

Label Name    Unique Values  Found In Streams
instance      27             27
pod           14             27
container     5              27
service_name  3              27
app           3              27
job           1              27
namespace     1              27
  • 27 streams for three applications in one hour. The driver is pod: every rollout creates new pod names and thus new streams. The old ones stop growing but stay in the index until retention.
  • instance has even more values because the container is part of it.
  • Imagine Alloy set status or path as a label: every new value multiplies the streams. With a user ID, there would be millions.

That’s why the rule in Loki is: labels describe where a line comes from. What it says stays in the text and is parsed at query time.

Save your progress? Sign up for free

Drill 4: The Ruler

The ruler already runs inside Loki but knows neither rules nor Alertmanager. Extend loki.yaml – here is the whole file:

cd ~/obs-lab
cat > loki.yaml <<'EOF'
deploymentMode: Monolithic

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  schemaConfig:
    configs:
      - from: "2026-01-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h
  limits_config:
    retention_period: 72h
  compactor:
    retention_enabled: true
    delete_request_store: filesystem
  rulerConfig:
    storage:
      type: local
      local:
        directory: /rules
    rule_path: /var/loki/rules-temp
    alertmanager_url: http://kps-alertmanager.monitoring:9093
    enable_api: true
    wal:
      dir: /var/loki/ruler-wal

singleBinary:
  replicas: 1
  persistence:
    size: 2Gi
  resources:
    requests: {cpu: 100m, memory: 128Mi}
    limits: {memory: 512Mi}

read: {replicas: 0}
write: {replicas: 0}
backend: {replicas: 0}
gateway: {enabled: false}
chunksCache: {enabled: false}
resultsCache: {enabled: false}
lokiCanary: {enabled: false}
test: {enabled: false}

sidecar:
  rules:
    folder: /rules/fake
EOF
helm upgrade loki oci://ghcr.io/grafana-community/helm-charts/loki --version 20.4.1 \
  -n monitoring -f loki.yaml --wait

Expected (shortened):

Pulled: ghcr.io/grafana-community/helm-charts/loki:20.4.1
...
STATUS: deployed
REVISION: 2
  • rulerConfig.storage tells the ruler that its rules are files under /rules. That’s where the sidecar from Day 7 writes to.
  • Without multi-tenancy, the only tenant in Loki is called fake. Rules live in a subfolder per tenant – hence folder: /rules/fake for the sidecar.
  • alertmanager_url points to the same Alertmanager as Prometheus.
  • enable_api allows querying the loaded rules.

Save your progress? Sign up for free

Drill 5: An Alert from the Log

faults.go writes faults enabled once at startup when a fault is configured. There is no metric for that – but there is a log line:

cat > notes-log-rules.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: notes-api-log-rules
  namespace: monitoring
  labels:
    loki_rule: "1"
data:
  notes-api.yaml: |
    groups:
      - name: notes-api.logs
        rules:
          - alert: NotesApiFaultsEnabled
            expr: |
              sum by (namespace) (count_over_time(
                {app="notes-api"} | json | msg="faults enabled" [10m])) > 0
            labels:
              severity: info
            annotations:
              summary: notes-api läuft mit eingebauten Störungen
EOF
kubectl apply -f notes-log-rules.yaml

Expected:

configmap/notes-api-log-rules created
  • The format is the same as in Prometheus: groups, rules, expr, labels, annotations. Only the query is LogQL.
  • The line appears once per pod in the log. [10m] keeps the alert active for ten minutes after the last start. After that it goes out, even if the fault is still configured.

After a minute, the ruler knows the rule:

curl -s localhost:30310/prometheus/api/v1/rules \
  | jq -c '.data.groups[].rules[] | [.name, .state, .health]'

Expected:

["NotesApiFaultsEnabled","inactive","ok"]

Under /prometheus/api/v1/rules, Loki answers in the same format as Prometheus. That’s why Grafana shows rules from both in the same place.

Now a small fault:

kubectl -n app set env deploy/notes-api NOTES_DELAY=50ms
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

After about a minute:

amtool alert query alertname=NotesApiFaultsEnabled

Expected:

Alertname              Starts At                Summary...
NotesApiFaultsEnabled  2026-11-05 09:19:23 UTC  notes-api läuft mit eingebauten Störu...

It wasn’t Prometheus that triggered the alert but Loki. For Alertmanager that makes no difference – alarm-echo gets it too.

Take back the fault:

kubectl -n app set env deploy/notes-api NOTES_DELAY-

Save your progress? Sign up for free

Checkpoint

kubectl -n app set env deploy/notes-api --list | grep LOG_FORMAT

Expected:

NOTES_LOG_FORMAT=json
NOTES_LOG_FORMAT=json

Once for notes-api, once for the init container migrate – set env changes all containers. JSON stays on from now on.

kubectl -n monitoring logs deploy/alarm-echo | grep '^{' \
  | jq -r '.json.alerts[].labels.alertname' | sort -u

Expected:

NotesApiDown
NotesApiErrors
NotesApiFaultsEnabled
NotesApiSlow

All alerts in this book so far, three from Prometheus, one from Loki.

Challenge

Write a second log rule NotesApiServerErrors: it should fire when notes-api writes more than 20 lines with level ERROR in five minutes. Trigger it with NOTES_ERROR_RATE=0.2.

Hint: A ConfigMap may contain several rules. Change the file and apply it again.

Solution

cat > notes-log-rules.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: notes-api-log-rules
  namespace: monitoring
  labels:
    loki_rule: "1"
data:
  notes-api.yaml: |
    groups:
      - name: notes-api.logs
        rules:
          - alert: NotesApiFaultsEnabled
            expr: |
              sum by (namespace) (count_over_time(
                {app="notes-api"} | json | msg="faults enabled" [10m])) > 0
            labels:
              severity: info
            annotations:
              summary: notes-api läuft mit eingebauten Störungen
          - alert: NotesApiServerErrors
            expr: |
              sum by (namespace) (count_over_time(
                {app="notes-api"} | json | level="ERROR" [5m])) > 20
            labels:
              severity: warning
            annotations:
              summary: "notes-api: {{ $value }} Fehlerzeilen in 5 Minuten"
EOF
kubectl apply -f notes-log-rules.yaml
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2

After two minutes:

amtool alert query alertname=NotesApiServerErrors

Expected:

Alertname             Starts At                Summary...
NotesApiServerErrors  2026-11-05 09:22:23 UTC  notes-api: 178 Fehlerzeilen in 5 Minut...

Take it back:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-

Metric or log? NotesApiErrors from Day 4 measures the same thing more precisely and more cheaply. Log rules pay off for events that no metric knows about.

Save your progress? Sign up for free

Enterprise Note

  • Define the log format: A common JSON schema for all applications – field names for time, level, message, trace ID – saves a special case in every query. OpenTelemetry defines the Semantic Conventions for that.
  • Structured metadata: Loki 3 can store fields like trace_id next to the line without turning them into labels. They are filterable but don’t increase cardinality.
  • Keep an eye on costs: logcli series --analyze-labels belongs in every regular review. A single wrong label can multiply the storage of a Loki.