Skip to content
Hands-on IT IT Drills

Day 7: Logs with Loki and Alloy

Goal: Alloy collects the logs of all pods, Loki stores them, and you query them with LogQL – on the command line and in Grafana. · Duration: approx. 75 min · Prerequisite: Day 6

Briefing

  • Loki stores logs the way Prometheus stores metrics: as streams with labels. Unlike a full-text search, Loki only indexes the labels, not the text. That makes it cheap to run – and requires you to narrow down by labels before you search in the text.
  • Alloy is Grafana’s collector. It finds the pods through the Kubernetes API, reads their logs and sends them to Loki with labels like namespace and pod.
  • LogQL is the query language: {namespace="app"} selects streams, after that stages like |= "POST" or | logfmt filter and parse the lines.
  • Logs turn into numbers too: count_over_time and rate work like in PromQL.
 Pods ──► Kubernetes API ──► Alloy ── push ──► Loki ◄── logcli (:30310)
                                                 ▲
                                               Grafana (data source Loki)

Drill 1: Installing Loki

The Loki chart is an OCI artifact in the GitHub Container Registry. Helm fetches it directly from there; a helm repo add isn’t needed:

cd ~/obs-lab
cat > loki.yaml <<'EOF'
deploymentMode: Monolithic

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  schemaConfig:
    configs:
      - from: "2026-01-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h
  limits_config:
    retention_period: 72h
  compactor:
    retention_enabled: true
    delete_request_store: filesystem

singleBinary:
  replicas: 1
  persistence:
    size: 2Gi
  resources:
    requests: {cpu: 100m, memory: 128Mi}
    limits: {memory: 512Mi}

read: {replicas: 0}
write: {replicas: 0}
backend: {replicas: 0}
gateway: {enabled: false}
chunksCache: {enabled: false}
resultsCache: {enabled: false}
lokiCanary: {enabled: false}
test: {enabled: false}
EOF
helm install loki oci://ghcr.io/grafana-community/helm-charts/loki --version 20.4.1 \
  -n monitoring -f loki.yaml --wait --timeout 10m

Expected (shortened):

Pulled: ghcr.io/grafana-community/helm-charts/loki:20.4.1
...
NAME: loki
NAMESPACE: monitoring
STATUS: deployed
  • deploymentMode: Monolithic: Loki runs as a single process. At large scale it is split into read, write and background paths – that’s why read, write and backend are explicitly set to 0.
  • auth_enabled: false turns off multi-tenancy. Otherwise Loki requires an X-Scope-OrgID header on every request.
  • schemaConfig defines how Loki stores index and data. tsdb and v13 are the current state; the date is the cutoff from which this schema applies.
  • retention_period: 72h with retention_enabled: the compactor deletes logs after three days – like retention: 3d in Prometheus.
  • A single process doesn’t need gateway, caches and canary.
kubectl -n monitoring get pods -l app.kubernetes.io/name=loki

Expected:

NAME     READY   STATUS    RESTARTS   AGE
loki-0   2/2     Running   0          51s

The second container is a sidecar like in Grafana. It collects rules for Loki – on Day 8 it gets work.

Save your progress? Sign up for free

Drill 2: Alloy Collects the Logs

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update grafana
cat > alloy.yaml <<'EOF'
controller:
  type: deployment

alloy:
  resources:
    requests: {cpu: 20m, memory: 64Mi}
    limits: {memory: 256Mi}
  configMap:
    content: |
      discovery.kubernetes "pods" {
        role = "pod"
      }

      discovery.relabel "pods" {
        targets = discovery.kubernetes.pods.targets
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          target_label  = "app"
        }
      }

      loki.source.kubernetes "pods" {
        targets    = discovery.relabel.pods.output
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki:3100/loki/api/v1/push"
        }
      }
EOF
helm install alloy grafana/alloy --version 1.15.0 -n monitoring -f alloy.yaml --wait

Expected (shortened):

"grafana" has been added to your repositories
...
NAME: alloy
NAMESPACE: monitoring
STATUS: deployed

The Alloy configuration is a chain of components. Each has a type and a name and passes its result on to the next:

  • discovery.kubernetes asks the API for all pods.
  • discovery.relabel turns the metadata (__meta_…) into the labels that land in Loki. Only these four – every additional label would be another index.
  • loki.source.kubernetes reads the pods’ logs through the Kubernetes API, just like kubectl logs. That’s why a single pod as a deployment is enough.
  • loki.write sends everything to the Service loki in the same namespace.

Save your progress? Sign up for free

Drill 3: logcli

Like Prometheus, Loki has a tool for the shell. So that logcli can reach Loki, Loki gets a fixed NodePort:

cat > loki-extern.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: loki-extern
  namespace: monitoring
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: loki
    app.kubernetes.io/component: single-binary
  ports:
    - port: 3100
      nodePort: 30310
EOF
kubectl apply -f loki-extern.yaml
curl -fsSLO https://github.com/grafana/loki/releases/download/\
v3.7.9/logcli-linux-amd64.zip
unzip -q logcli-linux-amd64.zip
mv logcli-linux-amd64 ~/.local/bin/logcli
rm logcli-linux-amd64.zip
echo 'export LOKI_ADDR=http://localhost:30310' >> ~/.bashrc
export LOKI_ADDR=http://localhost:30310
logcli labels --quiet

Expected:

app
container
instance
job
namespace
pod
service_name
  • LOKI_ADDR is the address that logcli reads on its own. --quiet suppresses the line in which logcli logs its request to Loki.
  • job and instance are set by loki.source.kubernetes itself, for example instance="app/notes-api-57f4f47769-727k6:notes-api".
  • You didn’t create service_name either. Loki derives it on ingestion from other labels, here from app – or unknown_service if none fits.
logcli labels --quiet app

Expected (shortened):

alarm-echo
alertmanager
alloy
grafana
...
last
loki
notes-api
postgres
prometheus
prometheus-node-exporter

Save your progress? Sign up for free

Drill 4: LogQL

The last three lines of notes-api:

logcli query --limit 3 --quiet '{namespace="app", app="notes-api"}'

Expected (shortened):

2026-11-05T09:08:58Z {instance="app/notes-api-57f4f47769-727k6:notes-api",
pod="notes-api-57f4f47769-727k6"} time=2026-11-05T09:08:58.263Z level=INFO msg=request
method=GET path=/notes status=200 duration_ms=3
...
  • {…} is the stream selector. It must narrow down at least one label – Loki never searches everything.
  • logcli only shows the labels in which the lines differ. It omits the common ones (app, namespace …).

Only the POST requests – a line filter:

logcli query --limit 2 --quiet -o raw '{app="notes-api"} |= "method=POST"'

Expected:

time=2026-11-05T09:08:57.495Z level=INFO msg=request method=POST ... status=201 ...
time=2026-11-05T09:08:56.497Z level=INFO msg=request method=POST ... status=201 ...

-o raw shows only the line itself. |= means “contains”, != “does not contain”, |~ allows regular expressions.

The lines of notes-api have the format key=value – logfmt. The parser turns them into labels you can filter by:

logcli query --limit 2 --quiet -o raw \
  '{app="notes-api"} | logfmt | duration_ms > 10'

Expected (maybe no line at all – then no request was slower):

time=2026-11-05T09:08:58.263Z level=INFO msg=request ... status=200 duration_ms=15
time=2026-11-05T09:08:54.488Z level=INFO msg=request ... status=200 duration_ms=20

duration_ms > 10 compares as a number. The labels from the parser only exist in this query; they are not in Loki’s index.

Save your progress? Sign up for free

Drill 5: Numbers from Logs

How many lines does each application write per second?

logcli instant-query --quiet \
  'sum by (app) (rate({namespace=~"app|monitoring"}[5m]))'

Expected (shortened, the numbers differ):

[
  ...
  {
    "metric": {
      "app": "notes-api"
    },
    "value": [
      1793869739.156,
      "4.83"
    ]
  },
  ...
  • notes-api writes one line per request, so about five per second. It doesn’t log /metrics and /healthz – otherwise every Prometheus scrape would be a log line.
  • last writes almost nothing: curl -sS only speaks up on errors, for example after a rollout of notes-api.
  • The result is JSON. For more than a glance, Grafana is the better place.

Save your progress? Sign up for free

Drill 6: Loki in Grafana

Grafana has a second sidecar for data sources. It reacts to ConfigMaps with the label grafana_datasource: "1":

cat > loki-datasource.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-datasource
  namespace: monitoring
  labels:
    grafana_datasource: "1"
data:
  loki.yaml: |
    apiVersion: 1
    datasources:
      - name: Loki
        uid: loki
        type: loki
        url: http://loki.monitoring:3100
EOF
kubectl apply -f loki-datasource.yaml

Expected:

configmap/loki-datasource created

The content is a Grafana provisioning file: the same format Grafana also reads from /etc/grafana/provisioning. After half a minute:

curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/datasources \
  | jq -c '.[] | [.name, .uid]'

Expected:

["Alertmanager","alertmanager"]
["Loki","loki"]
["Prometheus","prometheus"]

In the UI: Explore, data source Loki at the top, editor on Code, query {app="notes-api"} | logfmt | status >= 400, Run query. At the top right, Live switches to continuous output like kubectl logs -f – for all pods at once.

Save your progress? Sign up for free

Checkpoint

logcli query --limit 1 --quiet -o raw '{app="notes-api"} |= "msg=listening"' --since 24h

Expected (time and PID differ):

time=2026-11-05T09:04:59.584Z level=INFO msg=listening addr=:8080 version=0.8.0 pid=1

The startup line of a pod. Alloy fetched it after the fact: on first contact with a pod, it reads its log from the beginning.

kubectl -n monitoring get pods -l 'app.kubernetes.io/name in (loki,alloy)'

Expected:

NAME                     READY   STATUS    RESTARTS   AGE
alloy-758f96785b-lqndk   2/2     Running   0          112s
loki-0                   2/2     Running   0          2m50s

Challenge

Trigger errors as on Day 4 (NOTES_ERROR_RATE=0.2) and use logcli to find the error message notes-api writes for them. What does it say?

Hint: notes-api writes errors with level=ERROR. The line filter comes before the parser – it’s much cheaper.

Solution

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2
kubectl -n app rollout status deploy/notes-api
logcli query --limit 2 --quiet -o raw '{app="notes-api"} |= "level=ERROR"'

Expected (shortened):

time=... level=ERROR msg=request method=GET path=/notes status=500 duration_ms=0
time=... level=ERROR msg="notes failed" err="storage: request timed out"

Two lines per error: the cause from the handler and the access line, which also appears as ERROR for status 500. Take it back:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-

Save your progress? Sign up for free

Enterprise Note

  • Alloy as a DaemonSet: In the lab, Alloy reads the logs through the API. With many pods, that puts load on the API server. In production, Alloy runs on every node and reads the files under /var/log/pods directly.
  • Object storage: A production Loki stores data in S3, GCS or Azure Blob, not in the file system. Only then can it be split into read and write paths.
  • Go easy on labels: Rule of thumb for Loki: few labels with few values – environment, namespace, application. Everything else stays in the text and is parsed at query time. Day 8 shows why.