Day 7: Logs with Loki and Alloy
Goal: Alloy collects the logs of all pods, Loki stores them, and you query them with LogQL – on the command line and in Grafana. · Duration: approx. 75 min · Prerequisite: Day 6
Briefing
- Loki stores logs the way Prometheus stores metrics: as streams with labels. Unlike a full-text search, Loki only indexes the labels, not the text. That makes it cheap to run – and requires you to narrow down by labels before you search in the text.
- Alloy is Grafana’s collector. It finds the pods
through the Kubernetes API, reads their logs and sends them to Loki with
labels like
namespaceandpod. - LogQL is the query language:
{namespace="app"}selects streams, after that stages like|= "POST"or| logfmtfilter and parse the lines. - Logs turn into numbers too:
count_over_timeandratework like in PromQL.
Pods ──► Kubernetes API ──► Alloy ── push ──► Loki ◄── logcli (:30310)
▲
Grafana (data source Loki)
Drill 1: Installing Loki
The Loki chart is an OCI artifact in the GitHub Container Registry.
Helm fetches it directly from there; a helm repo add isn’t
needed:
cd ~/obs-lab
cat > loki.yaml <<'EOF'
deploymentMode: Monolithic
loki:
auth_enabled: false
commonConfig:
replication_factor: 1
storage:
type: filesystem
schemaConfig:
configs:
- from: "2026-01-01"
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
retention_period: 72h
compactor:
retention_enabled: true
delete_request_store: filesystem
singleBinary:
replicas: 1
persistence:
size: 2Gi
resources:
requests: {cpu: 100m, memory: 128Mi}
limits: {memory: 512Mi}
read: {replicas: 0}
write: {replicas: 0}
backend: {replicas: 0}
gateway: {enabled: false}
chunksCache: {enabled: false}
resultsCache: {enabled: false}
lokiCanary: {enabled: false}
test: {enabled: false}
EOF
helm install loki oci://ghcr.io/grafana-community/helm-charts/loki --version 20.4.1 \
-n monitoring -f loki.yaml --wait --timeout 10mExpected (shortened):
Pulled: ghcr.io/grafana-community/helm-charts/loki:20.4.1
...
NAME: loki
NAMESPACE: monitoring
STATUS: deployed
deploymentMode: Monolithic: Loki runs as a single process. At large scale it is split into read, write and background paths – that’s whyread,writeandbackendare explicitly set to 0.auth_enabled: falseturns off multi-tenancy. Otherwise Loki requires anX-Scope-OrgIDheader on every request.schemaConfigdefines how Loki stores index and data.tsdbandv13are the current state; the date is the cutoff from which this schema applies.retention_period: 72hwithretention_enabled: the compactor deletes logs after three days – likeretention: 3din Prometheus.- A single process doesn’t need gateway, caches and canary.
kubectl -n monitoring get pods -l app.kubernetes.io/name=lokiExpected:
NAME READY STATUS RESTARTS AGE
loki-0 2/2 Running 0 51s
The second container is a sidecar like in Grafana. It collects rules for Loki – on Day 8 it gets work.
Save your progress? Sign up for free
Drill 2: Alloy Collects the Logs
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update grafana
cat > alloy.yaml <<'EOF'
controller:
type: deployment
alloy:
resources:
requests: {cpu: 20m, memory: 64Mi}
limits: {memory: 256Mi}
configMap:
content: |
discovery.kubernetes "pods" {
role = "pod"
}
discovery.relabel "pods" {
targets = discovery.kubernetes.pods.targets
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
target_label = "app"
}
}
loki.source.kubernetes "pods" {
targets = discovery.relabel.pods.output
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}
EOF
helm install alloy grafana/alloy --version 1.15.0 -n monitoring -f alloy.yaml --waitExpected (shortened):
"grafana" has been added to your repositories
...
NAME: alloy
NAMESPACE: monitoring
STATUS: deployed
The Alloy configuration is a chain of components. Each has a type and a name and passes its result on to the next:
discovery.kubernetesasks the API for all pods.discovery.relabelturns the metadata (__meta_…) into the labels that land in Loki. Only these four – every additional label would be another index.loki.source.kubernetesreads the pods’ logs through the Kubernetes API, just likekubectl logs. That’s why a single pod as adeploymentis enough.loki.writesends everything to the Servicelokiin the same namespace.
Save your progress? Sign up for free
Drill 3: logcli
Like Prometheus, Loki has a tool for the shell. So that
logcli can reach Loki, Loki gets a fixed NodePort:
cat > loki-extern.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: loki-extern
namespace: monitoring
spec:
type: NodePort
selector:
app.kubernetes.io/name: loki
app.kubernetes.io/component: single-binary
ports:
- port: 3100
nodePort: 30310
EOF
kubectl apply -f loki-extern.yaml
curl -fsSLO https://github.com/grafana/loki/releases/download/\
v3.7.9/logcli-linux-amd64.zip
unzip -q logcli-linux-amd64.zip
mv logcli-linux-amd64 ~/.local/bin/logcli
rm logcli-linux-amd64.zip
echo 'export LOKI_ADDR=http://localhost:30310' >> ~/.bashrc
export LOKI_ADDR=http://localhost:30310
logcli labels --quietExpected:
app
container
instance
job
namespace
pod
service_name
LOKI_ADDRis the address thatlogclireads on its own.--quietsuppresses the line in whichlogclilogs its request to Loki.jobandinstanceare set byloki.source.kubernetesitself, for exampleinstance="app/notes-api-57f4f47769-727k6:notes-api".- You didn’t create
service_nameeither. Loki derives it on ingestion from other labels, here fromapp– orunknown_serviceif none fits.
logcli labels --quiet appExpected (shortened):
alarm-echo
alertmanager
alloy
grafana
...
last
loki
notes-api
postgres
prometheus
prometheus-node-exporter
Save your progress? Sign up for free
Drill 4: LogQL
The last three lines of notes-api:
logcli query --limit 3 --quiet '{namespace="app", app="notes-api"}'Expected (shortened):
2026-11-05T09:08:58Z {instance="app/notes-api-57f4f47769-727k6:notes-api",
pod="notes-api-57f4f47769-727k6"} time=2026-11-05T09:08:58.263Z level=INFO msg=request
method=GET path=/notes status=200 duration_ms=3
...
{…}is the stream selector. It must narrow down at least one label – Loki never searches everything.logclionly shows the labels in which the lines differ. It omits the common ones (app,namespace…).
Only the POST requests – a line
filter:
logcli query --limit 2 --quiet -o raw '{app="notes-api"} |= "method=POST"'Expected:
time=2026-11-05T09:08:57.495Z level=INFO msg=request method=POST ... status=201 ...
time=2026-11-05T09:08:56.497Z level=INFO msg=request method=POST ... status=201 ...
-o raw shows only the line itself. |= means
“contains”, != “does not contain”, |~ allows
regular expressions.
The lines of notes-api have the format key=value –
logfmt. The parser turns them into labels you can
filter by:
logcli query --limit 2 --quiet -o raw \
'{app="notes-api"} | logfmt | duration_ms > 10'Expected (maybe no line at all – then no request was slower):
time=2026-11-05T09:08:58.263Z level=INFO msg=request ... status=200 duration_ms=15
time=2026-11-05T09:08:54.488Z level=INFO msg=request ... status=200 duration_ms=20
duration_ms > 10 compares as a number. The labels
from the parser only exist in this query; they are not in Loki’s
index.
Save your progress? Sign up for free
Drill 5: Numbers from Logs
How many lines does each application write per second?
logcli instant-query --quiet \
'sum by (app) (rate({namespace=~"app|monitoring"}[5m]))'Expected (shortened, the numbers differ):
[
...
{
"metric": {
"app": "notes-api"
},
"value": [
1793869739.156,
"4.83"
]
},
...
- notes-api writes one line per request, so about five per second. It
doesn’t log
/metricsand/healthz– otherwise every Prometheus scrape would be a log line. lastwrites almost nothing:curl -sSonly speaks up on errors, for example after a rollout of notes-api.- The result is JSON. For more than a glance, Grafana is the better place.
Save your progress? Sign up for free
Drill 6: Loki in Grafana
Grafana has a second sidecar for data sources. It reacts to
ConfigMaps with the label grafana_datasource: "1":
cat > loki-datasource.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-datasource
namespace: monitoring
labels:
grafana_datasource: "1"
data:
loki.yaml: |
apiVersion: 1
datasources:
- name: Loki
uid: loki
type: loki
url: http://loki.monitoring:3100
EOF
kubectl apply -f loki-datasource.yamlExpected:
configmap/loki-datasource created
The content is a Grafana provisioning file: the same
format Grafana also reads from /etc/grafana/provisioning.
After half a minute:
curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/datasources \
| jq -c '.[] | [.name, .uid]'Expected:
["Alertmanager","alertmanager"]
["Loki","loki"]
["Prometheus","prometheus"]
In the UI: Explore, data source
Loki at the top, editor on Code, query
{app="notes-api"} | logfmt | status >= 400, Run
query. At the top right, Live switches to
continuous output like kubectl logs -f – for all pods at
once.
Save your progress? Sign up for free
Checkpoint
logcli query --limit 1 --quiet -o raw '{app="notes-api"} |= "msg=listening"' --since 24hExpected (time and PID differ):
time=2026-11-05T09:04:59.584Z level=INFO msg=listening addr=:8080 version=0.8.0 pid=1
The startup line of a pod. Alloy fetched it after the fact: on first contact with a pod, it reads its log from the beginning.
kubectl -n monitoring get pods -l 'app.kubernetes.io/name in (loki,alloy)'Expected:
NAME READY STATUS RESTARTS AGE
alloy-758f96785b-lqndk 2/2 Running 0 112s
loki-0 2/2 Running 0 2m50s
Challenge
Trigger errors as on Day 4 (NOTES_ERROR_RATE=0.2) and
use logcli to find the error message notes-api writes for
them. What does it say?
Hint: notes-api writes errors with
level=ERROR. The line filter comes before the parser – it’s
much cheaper.
Solution
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2
kubectl -n app rollout status deploy/notes-api
logcli query --limit 2 --quiet -o raw '{app="notes-api"} |= "level=ERROR"'Expected (shortened):
time=... level=ERROR msg=request method=GET path=/notes status=500 duration_ms=0
time=... level=ERROR msg="notes failed" err="storage: request timed out"
Two lines per error: the cause from the handler and the access line,
which also appears as ERROR for status 500. Take it
back:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-Save your progress? Sign up for free
Enterprise Note
- Alloy as a DaemonSet: In the lab, Alloy reads the
logs through the API. With many pods, that puts load on the API server.
In production, Alloy runs on every node and reads the files under
/var/log/podsdirectly. - Object storage: A production Loki stores data in S3, GCS or Azure Blob, not in the file system. Only then can it be split into read and write paths.
- Go easy on labels: Rule of thumb for Loki: few labels with few values – environment, namespace, application. Everything else stays in the text and is parsed at query time. Day 8 shows why.