Day 5: Grafana
Goal: You find your way around Grafana, use Explore and the bundled dashboards, and build a first dashboard of your own for notes-api. · Duration: approx. 60 min · Prerequisite: Day 4
Briefing
- Grafana draws what others store. It has no data of its own but queries data sources: Prometheus for metrics, from Day 7 Loki for logs, from Day 10 Tempo for traces.
- Explore is the place to try things out: one query, one graph, nothing saved.
- A dashboard consists of panels. Every panel has one or more queries and a visualization – time series, single stat, table.
- kube-prometheus-stack brings Grafana fully wired up: data sources and a good two dozen dashboards for Kubernetes are already there.
Browser ──► :30300 Grafana ──► data source Prometheus ──► kps-prometheus:9090
└────► data source Alertmanager ─► kps-alertmanager:9093
Drill 1: Logging In
Grafana answers whether it’s healthy even without logging in:
curl -s localhost:30300/api/healthExpected:
{
"database": "ok",
"version": "13.3.0",
"commit": "6f8ce3c693b493043ebcd4787566d2b6b118b0f0"
}
Open http://<VM-IP>:30300 in the browser and log
in: user admin, password lab-grafana-pw-123 –
the password from kps.yaml. Grafana doesn’t ask for a new
password because the chart set it.
Save your progress? Sign up for free
Drill 2: The Data Sources
Everything the UI shows is also available through the HTTP API. With
user and password via -u:
curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/datasources \
| jq -c '.[] | [.name, .uid, .url]'Expected:
["Alertmanager","alertmanager","http://kps-alertmanager.monitoring:9093/"]
["Prometheus","prometheus","http://kps-prometheus.monitoring:9090/"]
- Grafana talks to Prometheus through the Service in the cluster, not through the NodePort.
- The uid is the fixed name of a data source. Dashboards refer to it – that becomes important on Day 6.
- A helper container in the Grafana pod created the data sources. It reads ConfigMaps with a specific label and passes them on to Grafana. That’s why the pod has three containers.
In the UI you find them under Connections → Data sources.
Save your progress? Sign up for free
Drill 3: Explore
Click Explore in the menu on the left. At the top,
the data source is Prometheus.
- Switch the editor on the right from Builder to Code.
- Enter the recording rule from Day 3:
sum by (method) (job:notes_http_requests:rate5m) - Click Run query (or press
Shift+Enter).
Expected: two lines, GET just above 4 and
POST at 1, below them a table with the same values.
- At the top right you set the time range. Choose Last 30 minutes: the dips and spikes from Day 4 are visible – errors, pods at zero, delay.
- Under Options → Legend,
{{method}}turns{method="GET"}into a plainGET. - Explore doesn’t remember anything. What you want to keep goes into a dashboard.
Save your progress? Sign up for free
Drill 4: Bundled Dashboards
curl -s -u admin:lab-grafana-pw-123 \
'localhost:30300/api/search?type=dash-db&query=Kubernetes' | jq -r '.[].title'Expected (shortened):
Kubernetes / API server
Kubernetes / Compute Resources / Multi-Cluster
Kubernetes / Compute Resources / Cluster
Kubernetes / Compute Resources / Namespace (Pods)
...
Kubernetes / Persistent Volumes
In the UI, open Dashboards and there
Kubernetes / Compute Resources / Namespace (Pods). At
the top, choose the value app for
namespace.
Expected: panels for CPU and memory, split by pod.
postgres-0 needs the most CPU, the two notes-api pods share
the rest.
Compare with what Kubernetes itself reports:
kubectl -n app top podsExpected (the numbers differ):
NAME CPU(cores) MEMORY(bytes)
last-6fdd4b976d-47f97 14m 3Mi
notes-api-57f4f47769-frk2p 13m 7Mi
notes-api-57f4f47769-s66pc 11m 7Mi
postgres-0 33m 64Mi
The same orders of magnitude. kubectl top asks the
metrics-server of k3s, the dashboard asks Prometheus for the cAdvisor
metrics. Both come from the kubelet.
Save your progress? Sign up for free
Drill 5: A Dashboard of Your Own
- Dashboards → New → New dashboard, then Add visualization.
- Choose the data source Prometheus.
- Switch the query editor to Code and enter:
sum by (method) (job:notes_http_requests:rate5m) - Set Legend to Custom and enter
{{method}}. - In the options pane on the right: Title
Anfragen pro Sekunde, under Standard options → Unitrequests/sec (rps). - At the top, Save dashboard, title
notes-api, save.
A second panel for the response time:
- Add → Visualization.
- Query:
job:notes_http_request_duration_seconds:p95_5m, legendp95. - Title
Antwortzeit p95, Unitseconds (s). - Under Thresholds, change the red value from
80to0.25and chooseAs linesfor Show thresholds. - Save.
Expected: two panels. The line in the second is at a few milliseconds; the red line at 250 ms is the same limit as in the NotesApiSlow alert.
Save your progress? Sign up for free
Drill 6: The Dashboard via the API
curl -s -u admin:lab-grafana-pw-123 \
'localhost:30300/api/search?type=dash-db&query=notes-api' | jq '.[0] | {title, uid}'Expected (the uid differs):
{
"title": "notes-api",
"uid": "al6g5q"
}
The uid is the fixed name of the dashboard; it also
appears in its address in the browser:
http://<VM-IP>:30300/d/al6g5q/notes-api.
You see the content in the UI: in the dashboard at the top, Export → Export as JSON. A dashboard is nothing more than a JSON document with panels, queries and settings. If you click it together, it only lives in Grafana’s database. Day 6 shows how it gets into Git.
Save your progress? Sign up for free
Checkpoint
curl -s -u admin:lab-grafana-pw-123 \
'localhost:30300/api/search?type=dash-db&query=notes-api' | jq -r '.[].title'Expected:
notes-api
curl -s -u admin:lab-grafana-pw-123 \
'localhost:30300/api/search?type=dash-db' | jq lengthExpected:
26
25 bundled ones and your own.
Challenge
Add a panel Fehlerquote to the dashboard that shows
the share of 5xx responses in percent. Trigger a fault with
NOTES_ERROR_RATE=0.1 and watch.
Hint: The recording rule has the label
code. Percent (0.0-1.0) fits as the unit.
Solution
Query in the new panel:
sum(job:notes_http_requests:rate5m{code=~"5.."})
/ sum(job:notes_http_requests:rate5m)
Unit Percent (0.0-1.0), title
Fehlerquote, save. Then:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.1Expected: within five minutes the line rises to about 9 percent – not
10, because /healthz and /metrics never fail
and count in the denominator. Without a fault, the panel shows
No data: if there is no 5xx response, there is no time
series with code=~"5.." either.
Take it back:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-Save your progress? Sign up for free
Enterprise Note
- Login via SSO: In production, nobody logs in as
admin. Grafana speaks OIDC, SAML and LDAP and maps groups to roles: Viewer, Editor, Admin. - Folders and permissions: Dashboards live in folders, permissions are attached to the folder. Every team gets its own.
- “No data” is a statement: An empty panel can mean
“no errors” or “no data”. With
or vector(0)at the end of the query, the panel shows 0 instead of nothing – but also when notes-api isn’t being scraped at all.